Databricks Engineer

Role Overview

We are seeking a skilled and experienced Databricks Data Engineer to join our data platform team. In this role, you will design, build, and optimize high-throughput batch and real-time data pipelines on the Databricks Lakehouse platform. You will implement best practices using Delta Lake, Apache Spark, and Unity Catalog to ensure our data infrastructure is scalable, secure, and cost-effective.

Key Responsibilities

• Architecture and deploy robust data pipelines using Apache Spark, PySpark, and Spark SQL to ingest data from diverse sources (APIs, relational databases, cloud storage, event streams).
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake to deliver clean, transactional, and analytics-ready datasets.
• Automate batch and streaming workflows using Databricks Lakeflow Jobs, Workflows, or Apache Airflow.
• Set up fine-grained access control, security policies, and data lineage tracking using Unity Catalog.
• Optimize Spark jobs, query performance, and compute cluster configurations (autoscaling, caching, partitioning) to maintain performance while minimizing cloud costs.
• Partner with Data Scientists, Business Intelligence Engineers, and product teams to translate business requirements into efficient data models. Enforce unit testing and CI/CD pipelines for data engineering code.

Required Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or equivalent practical experience.
• 3+ years of hands-on experience in data engineering, with at least 2+ years actively building on Databricks.
• Strong proficiency in Python (PySpark) and SQL (Scala is a plus).
• Core Technologies:
• Deep experience with Apache Spark and Delta Lake.
• Hands-on experience with cloud platforms (AWS, Azure, or GCP).
• Familiarity with Unity Catalog for governance and Databricks Workflows for orchestration.
• Strong experience with Git, CI/CD pipelines, and writing testable, modular code.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience with real-time streaming engines (Structured Streaming, Apache Kafka, Kinesis).
• Knowledge of dbt (data build tool) integrated with Databricks.
• Familiarity with machine learning operations (MLflow) and vector databases on Databricks.