Technical Lead - Data Engineer ( Databricks Azure)

We are seeking a highly skilled and experienced Data Engineering with strong expertise in Microsoft Azure data services, enterprise data integration, and modern data lake architectures. In this role, you will lead the design, development, and optimization of scalable data pipelines responsible for ingesting and transforming data from enterprise systems including Microsoft platforms, SAP, CRM systems, retail systems, and external third-party data sources into a centralized data platform to support downstream applications, analytics, AI, and agentic use cases.

You will play a critical role in building a robust, high-performance data platform using Azure-native services, ensuring data quality, reliability, and near real-time availability for business operations. The ideal candidate will have hands-on experience building Medallion Architecture (Bronze, Silver, Gold) data lakes, integrating diverse enterprise data sources, and enabling AI-powered solutions using the Microsoft ecosystem.

Responsibilities

• Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems including Microsoft applications, SAP, CRM platforms, retail systems, and external third-party sources into Azure-based data platforms.
• Design, build, and maintain enterprise-scale Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure Data Lake Storage, Microsoft Fabric, Databricks, or similar technologies.
• Develop and optimize data processing workflows using Azure Databricks, Apache Spark / PySpark, Azure Data Factory, Microsoft Fabric, and other Azure-native services.
• Create denormalized, API-ready, analytics-ready, and AI-ready data models aligned with downstream applications, microservices, reporting, and agentic AI consumption patterns.
• Implement idempotent processing, CDC merge (upsert) strategies, incremental processing, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
• Leverage Azure Data Lake Storage Gen2, Azure SQL, Azure Synapse Analytics, Microsoft Fabric, Cosmos DB, and related services for efficient storage and querying of structured and semi-structured data.
• Implement data ingestion patterns (batch, streaming, and event-driven) using tools such as Azure Event Hubs, Azure Functions, Logic Apps, Service Bus, and Azure Data Factory.
• Lead enterprise data integration initiatives across Microsoft platforms, SAP, SaaS applications, APIs, partner systems, and external data providers.
• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, reliability, and cost-effectiveness.
• Define and enforce data governance, data quality, metadata management, and data lineage standards across the data platform.
• Collaborate with DevOps teams to design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.
• Support AI and agent-enabled solutions leveraging Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, and related Microsoft AI technologies.
• Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.
• Collaborate with application, microservices, analytics, and AI teams to ensure seamless integration with enterprise data platforms via APIs, event streams, and data products.

• 5+ years of experience in data engineering, with at least 2+ years in a lead role.
• Strong hands-on experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Microsoft Fabric, and Azure SQL.
• Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze, Silver, Gold) and modern lakehouse solutions.
• Proficiency in PySpark / Spark for large-scale data processing and optimization.
• Experience designing and implementing ODS, Lakehouse, or operational data layers using technologies such as Cosmos DB, Azure SQL, Microsoft Fabric, Databricks, or similar platforms.
• Strong expertise in ETL/ELT design patterns, enterprise data integration, data ingestion, and transformation pipelines.
• Experience integrating data from Microsoft platforms, SAP systems, CRM applications, REST APIs, SaaS applications, and external third-party data sources.
• Hands-on experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or similar).
• Good understanding of data governance, data quality, metadata management, and data lineage frameworks.
• Experience supporting microservices, analytics platforms, AI solutions, and agentic architectures with enterprise data platforms.
• Exposure to Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, or related Microsoft AI technologies is highly desirable.
• Strong problem-solving skills and ability to optimize complex data workflows.
• Excellent communication and stakeholder management skills.
• Ability to work in a fast-paced, agile environment and manage multiple priorities.