Senior Data Engineer

No third party recruiters please

Must be able to work W2

Job Summary:

We are seeking an experienced Data Engineer with 6–10 years of expertise in designing, building, and maintaining scalable, high-performance data pipelines and processing frameworks. The ideal candidate will have strong hands-on experience with orchestration using Apache Airflow and distributed data processing with Apache Spark (EMR). This role requires a solid understanding of big data architecture, data engineering best practices, and a commitment to delivering efficient, reliable, and maintainable data solutions that align with both business and technical requirements.

________________________________________

Roles and Responsibilities:

• Design, build, and manage scalable and reliable data pipelines using Apache Airflow.

• Develop and optimize large-scale data processing workflows using Apache Spark, including both batch and Structured Streaming.

• Collaborate with data architects, analysts, and business stakeholders to translate data requirements into efficient engineering solutions.

• Ensure the quality, performance, and security of data processes and systems.

• Monitor, troubleshoot, and optimize data workflows and job executions.

• Document solutions, workflows, and technical designs.

________________________________________

Qualifications Required:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.

• 6–10 years of experience in data engineering or related roles.

• Strong experience with Apache Airflow for data orchestration and workflow management.

• Proven expertise in building and tuning distributed data processing applications using Apache Spark (PySpark), including both Structured Streaming and Batch.

• Experience with AWS cloud-based data ecosystems, particularly Athena and Redshift.

• Proficient in SQL (Athena version) and experienced in working with large datasets from various sources (structured and unstructured).

• Experience with data lakes, data warehouses, and batch/streaming data architectures.

• Familiarity with CI/CD pipelines, version control, and DevOps practices in a data engineering context.

• Strong problem-solving and communication skills, with the ability to work both independently and collaboratively.

________________________________________

Preferred Qualifications:

• Familiarity with additional big data technologies (e.g., Hadoop, Kafka).

• Experience working in Agile/Scrum environments.

• Knowledge of data quality frameworks and validation engines.

• Experience with data catalog tools.