Back to search
InFynd Linkedin · Posted 28d ago

Data Engineer

India

Linkedin
Continue to application Add your email once, then Caio opens the original posting.

Indexed description

  • Design and implement scalable batch and streaming data pipelines using Azure Databricks, Delta Live Tables, and Apache Spark.
  • Build and maintain ETL/ELT workflows orchestrated through Azure Data Factory and Databricks Workflows.
  • Develop reusable and modular pipeline components following software engineering best practices.
  • Architect and manage Lakehouse solutions using Delta Lake across Bronze, Silver, and Gold layers.
  • Design and enforce data models, schemas, and governance policies using Unity Catalog.
  • Optimize storage, partitioning, and query performance for large-scale datasets on ADLS Gen2.
  • Manage Databricks clusters, compute policies, and job scheduling.
  • Implement Infrastructure as Code (IaC) using Terraform or ARM templates.
  • Integrate Databricks with Azure services including Synapse, Event Hubs, Key Vault, and Azure DevOps.

Requirements

  • Strong proficiency in PySpark, Python, and SQL for Big Data processing.
  • Hands-on experience with Delta Lake, Delta Live Tables (DLT), and Medallion Architecture.
  • Strong experience with Azure Data Services including:
    • Azure Data Lake Storage Gen2 (ADLS Gen2)
    • Azure Data Factory (ADF)
    • Azure Synapse Analytics
    • Azure Event Hubs
  • Experience with Databricks Unity Catalog for data governance and access control.
  • Experience implementing CI/CD pipelines using Azure DevOps or GitHub Actions for Databricks deployments.
  • Strong understanding of distributed computing concepts, Spark optimization, partitioning, and performance tuning.
  • Experience with streaming data processing using Structured Streaming, Kafka, or Event Hubs.
Free. 20 seconds. No password. See every match in this search.

Create a free Caio profile to unlock more results and save your role and location preferences.

Unlock free search
Want help applying to roles like this? Search Caio for free. If CV tailoring and application tracking get heavy, Full Caio Agent adds a human specialist.
View Full Agent