Data Engineer – Intermediate
Indexed description
Job Title: Data Engineer - Intermediate
Location: Manhattan West, NY - Onsite
Job Description:
We are seeking a skilled Data Engineer to design, build, and manage scalable ETL pipelines supporting a centralized data lake and Snowflake data warehouse. The role focuses on automating data ingestion, transformation, and aggregation workflows to enable reliable analytics and data-driven decision-making.
Key Responsibilities
- Design, develop, and maintain robust ETL pipelines for ingesting data into the enterprise data lake and Snowflake environment.
- Automate data processing, aggregation, and analytical workflows to improve data availability and performance.
- Implement and manage orchestration and scheduling of data pipelines using Control‐M and Apache Airflow.
- Develop scalable data transformation logic using PySpark and Apache Spark (Java).
- Work with large, structured and semi-structured datasets on AWS infrastructure.
- Ensure data quality, integrity, and reliability across data pipelines.
- Optimize data pipelines for performance, cost, and scalability.
- Collaborate with analytics, data science, and business teams to understand data requirements.
- Monitor, troubleshoot, and resolve pipeline failures and performance bottlenecks.
- Follow best practices for data engineering, security, and documentation.
- Strong experience with data lake architectures and large-scale data processing.
- Hands-on experience with AWS services (e.g., S3, EC2, EMR, Glue, or related).
- Proven expertise in building ETL pipelines for analytics and reporting use cases.
- Solid working knowledge of Snowflake, including data loading, transformations, and performance optimization.
- Experience with workflow automation and scheduling tools such as Control‐M and Apache Airflow.
- Proficiency in PySpark for distributed data processing.
- Strong programming experience with Apache Spark using Java.
- Good understanding of data modeling, partitioning, and performance tuning concepts.
- Exposure to CI/CD practices for data pipelines.
- Experience working in Agile or DevOps environments.
Create a free Caio profile to unlock more results and save your role and location preferences.
Unlock free search