Back to search
Techgene Solutions Linkedin · Posted 2d ago

Sr. Data Engineer

Pasadena, California, United States

Linkedin
Continue to application Add your email once, then Caio opens the original posting.

Indexed description

Role : Sr. Data Engineer

Location: Pasadena, CA

Work Arrangement: Hybrid – 3 Days/Week Onsite

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, and Python to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience building ETL/ELT pipelines, data processing frameworks, and data lake/lakehouse solutions using Databricks and cloud technologies.

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
  • Develop ETL/ELT workflows to ingest, transform, cleanse, and integrate data from multiple sources.
  • Build and optimize data processing jobs using PySpark and Spark SQL.
  • Work extensively with Databricks Lakehouse, Delta Lake, notebooks, workflows, and clusters.
  • Develop reusable Python modules and frameworks for data processing and automation.
  • Implement data quality checks, validation, error handling, and monitoring within data pipelines.
  • Optimize Spark jobs, including partitioning, caching, joins, and performance tuning.
  • Work with Delta Lake for data storage, transformation, versioning, and incremental processing.
  • Integrate data from relational databases, APIs, files, cloud storage, and other enterprise data sources.
  • Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders to understand data requirements.
  • Implement CI/CD and source-control practices for data engineering code.
  • Troubleshoot production data pipeline failures and perform root cause analysis (RCA).
  • Ensure data security, governance, lineage, and compliance requirements are followed.
  • Participate in design discussions, code reviews, testing, deployment, and production support.

Required Skills

  • Strong hands-on experience with Databricks
  • Strong PySpark / Apache Spark experience
  • Strong Python programming skills
  • Experience developing ETL/ELT pipelines
  • Strong SQL skills
  • Experience with Delta Lake
  • Experience with data lake/lakehouse architecture
  • Experience with Spark performance tuning and optimization
  • Experience working with large-volume datasets
  • Strong understanding of data modeling and data engineering concepts
  • Experience with Git and CI/CD
  • Experience with cloud platforms such as AWS, Azure, or GCP


Free. 20 seconds. No password. See every match in this search.

Create a free Caio profile to unlock more results and save your role and location preferences.

Unlock free search
Want help applying to roles like this? Search Caio for free. If repetitive applications get heavy, Managed Job Search adds supervised execution for $99/month.
View Managed Job Search