Data Engineer – Databricks
Indexed description
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
- Work extensively with SQL and Python for data transformation, analysis, and automation.
- Build and optimize data processing workflows on cloud-based analytic data stores such as Databricks.
- Develop and manage data ingestion from multiple structured and unstructured data sources.
- Ensure data quality, reliability, integrity, and performance across data platforms.
- Work with RDBMS systems and optimize complex SQL queries.
- Collaborate with cross-functional teams including analytics, business, and application teams.
- Monitor and troubleshoot production data pipelines and workflows.
- Implement best practices for data governance, security, and scalability.
- Strong hands-on experience with Databricks
- Expertise in SQL and Python
- Strong experience in ETL/ELT development
- Hands-on experience with Apache Spark
- Good understanding of RDBMS concepts
- Experience working with Analytical Data Stores/Data Warehousing
- Strong understanding of data quality, validation, and reliability frameworks
- Experience in performance tuning and optimization of data pipelines
- Experience with cloud platforms like AWS, Azure, or GCP
- Knowledge of Delta Lake, Data Lake architecture
- Exposure to CI/CD and DevOps practices in data engineering
- Experience with orchestration tools like Airflow or ADF
- Bachelor’s degree in Computer Science, Information Technology, or related field
- Strong analytical and problem-solving skills
- Good communication and collaboration abilities
Create a free Caio profile to unlock the full index and keep your job-search signal for future recommendations.
Unlock free search