Strong Middle Data Engineer

Strong Middle Data Engineer – Customer Platform & Data

Position Overview

We are seeking Strong Middle Data Engineers with solid experience in distributed data systems, Spark-based data processing, and production-grade data platform engineering.

In this role, you will build, scale, maintain, and improve large-scale data platforms supporting critical initiatives across customer data, data pipeline expansion, compliance-related engineering, new data source integration, and platform modernization.

The ideal candidate is a hands-on engineer who can work independently, take ownership of technical solutions, contribute to system design and implementation, and operate effectively in a highly automated development environment.

The engineering organization is evolving toward a spec-led and agentic development model, where engineers increasingly focus on defining requirements, designing data models, creating technical specifications, reviewing AI-generated code, and ensuring overall solution quality.

Key Responsibilities

• Develop, maintain, and optimize batch and streaming data pipelines using Scala and Apache Spark.
• Build and support Kafka-based data processing and streaming solutions.
• Create and maintain Apache Airflow DAGs, workflows, and data processing schedules.
• Design and implement data models, schemas, mappings, and validation checks.
• Develop scalable ETL/ELT pipelines using SQL and distributed data processing technologies.
• Monitor pipeline health, investigate failures, and troubleshoot production issues.
• Support deployments and ensure reliable operation of production data platforms.
• Implement data quality and validation controls to ensure accurate and reliable data.
• Review and validate AI-generated code and proposed implementations.
• Collaborate with senior engineers, product teams, and data consumers to deliver technical solutions.
• Contribute to system design, technical specifications, and modernization initiatives.
• Take ownership of assigned systems and components, including reliability, maintainability, and performance.

Required Qualifications

• 3+ years of professional Data Engineering experience.
• Strong hands-on experience with Scala and Apache Spark.
• Experience working with batch processing and distributed data systems.
• Practical experience with Kafka, Kafka Streams, Flink, or similar streaming technologies.
• Hands-on experience with Apache Airflow, including DAG development and workflow management.
• Strong knowledge of SQL, data modeling, and ETL/ELT development.
• Experience with software engineering practices including:
• Automated testing
• Code reviews
• Git/version control
• CI/CD
• Production deployments
• Strong troubleshooting and production support capabilities.
• Ability to work independently and take ownership of technical solutions.
• Good communication and collaboration skills.
• Ability to understand technical requirements and translate them into reliable data engineering solutions.

Nice-to-Have Qualifications

• Hands-on Apache Flink experience.
• Experience with Cassandra, DynamoDB, ScyllaDB, or other NoSQL databases.
• Experience working with customer data, clickstream, loyalty, booking, or transactional data.
• Experience with GitHub Copilot, Claude, Cursor, or other AI-assisted development tools.
• Understanding of data quality monitoring, observability, and pipeline health monitoring.
• Experience with specification-driven or spec-to-code development.
• Familiarity with agentic development frameworks and engineering practices.

Success Profile

The successful candidate is a strong middle-level Data Engineer who can work with limited supervision while knowing when to involve senior engineers.

You should be able to demonstrate:

• Practical experience building and maintaining Spark-based data pipelines.
• Solid Scala development experience in production.
• Understanding of distributed data processing and batch workloads.
• Hands-on experience with Kafka or comparable streaming technologies.
• Ability to build and manage Airflow DAGs and production workflows.
• Strong SQL, ETL/ELT, and data modeling skills.
• Ability to troubleshoot pipeline failures and production data issues.
• Experience participating in code reviews, testing, CI/CD, and deployments.
• Ability to review AI-generated code critically and identify incorrect, inefficient, or unsafe implementations.
• An ownership mindset toward data quality, reliability, maintainability, and delivery.
• Ability to communicate effectively with senior engineers, product teams, and data consumers.

Engineering Environment

You will work in a modern, highly automated data engineering environment using technologies including:

Scala | Apache Spark | Kafka | Kafka Streams | Apache Flink | Apache Airflow | SQL | NoSQL | CI/CD | AI-Assisted Development

The team is moving toward a spec-led, agentic engineering model, giving engineers greater responsibility for requirements definition, technical specifications, data modeling, solution design, and validation of AI-assisted implementations.

Why This Position?

• Work on large-scale data platforms supporting high-priority business initiatives.
• Gain exposure to complex customer, transactional, compliance, and operational data.
• Work with modern distributed data technologies including Spark, Scala, Kafka, and Airflow.
• Participate in the modernization and expansion of enterprise data platforms.
• Develop skills in AI-assisted and agentic software development.
• Collaborate closely with experienced senior engineers while owning meaningful technical components.
• Work in an engineering culture focused on automation, quality, reliability, and continuous improvement.