Senior Data Engineer

Responsibilities

• Maintain and improve the ingestion pipeline to reliably deliver billions of events daily within defined SLAs
• Provide support for all teams in building and optimizing their complex pipelines
• Work closely with other teams to identify pain points and problems around the platform
• Develop new tools and frameworks to improve the data platform
• Facilitate a company-wide, data-driven culture
• Work in close collaboration with data scientists and data analysts to help support their work going into production
• Work on large-scale projects such as building an ML platform and streaming use cases
• Set up best practices and processes around software and data development

Requirements

Mandatory

• Experience working with an orchestration tool (e.g., Airflow)
• Experience working with large data sets
• Experience building complex ETL pipelines
• Experience working with cloud providers such as GCP
• Strong programming skills in Spark with Scala and Python
• Experience working with CI/CD tools like Jenkins and Git
• Strong understanding of software engineering practices and principles
• Excellent problem-solving and communication skills
• Self-motivated with a proven ability to take initiative to own and solve problems as they come up

Nice-to-have

• Experience working with messaging systems like Kafka
• Knowledge of Kubernetes
• Hands-on experience with any streaming platform
• Experience managing a data warehouse in BigQuery or Redshift