Back to search
ATC Linkedin · Posted yesterday

Data Engineer

New York, United States

Linkedin
Continue to application Add your email once, then Caio opens the original posting.

Indexed description

Job Summary

We are seeking a Data Engineer with 5+ years of experience in designing, developing, and optimizing scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Databricks, Apache Spark, and ETL/ELT development, along with hands-on experience in at least one cloud platform (AWS, Azure, or Google Cloud Platform).


Key Responsibilities

  • Design, develop, and maintain scalable ETL/ELT data pipelines.
  • Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
  • Develop and maintain data lakes and cloud-based data warehouses.
  • Create scalable batch and real-time data processing solutions.
  • Optimize data pipelines for performance, reliability, and scalability.
  • Develop reusable data engineering frameworks and automation solutions.
  • Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
  • Implement data quality, governance, monitoring, and security best practices.
  • Troubleshoot production issues and continuously improve data platform performance.
  • Follow DevOps and CI/CD best practices for data engineering projects.


Required Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
  • 5+ years of experience as a Data Engineer.
  • Strong programming skills in Python.
  • Advanced SQL proficiency.
  • Hands-on experience with Databricks.
  • Strong experience with Apache Spark (PySpark preferred).
  • Experience building and maintaining ETL/ELT pipelines.
  • Experience with Apache Airflow or similar workflow orchestration tools.
  • Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
  • Hands-on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
  • Experience with Git, Docker, and CI/CD pipelines.
  • Knowledge of relational and NoSQL databases.


Preferred Qualifications

  • Experience with Kafka or other streaming platforms.
  • Experience with Delta Lake.
  • Familiarity with Apache Iceberg or Apache Hudi.
  • Experience with Infrastructure as Code (Terraform or CloudFormation).
  • Knowledge of data governance and data quality frameworks.
  • Exposure to DevOps and MLOps practices.


Free. 20 seconds. No password. See every match in this search.

Create a free Caio profile to unlock more results and save your role and location preferences.

Unlock free search
Want help applying to roles like this? Search Caio for free. If CV tailoring and application tracking get heavy, Full Caio Agent adds a human specialist.
View Full Agent