Databricks PlatformLead

Job Title: Senior Data Platform Engineer (Mgr/Sr Mgr grade)

Location: Navi Mumbai

Function: Information Management / IT

Key Accountabilities

1. Platform Strategy & Technical Leadership

• Own the enterprise data platform strategy and roadmap, guiding the transition from legacy platforms to Microsoft Fabric, Databricks and/or Snowflake.
• Act as the primary technical decision-maker for data platform tools, standards and architecture choices.
• Lead architecture and design reviews, establishing engineering quality standards.
• Represent data platform engineering in leadership forums and vendor discussions.
• Stay current with data integration, automation and emerging data platform technologies and apply them where they deliver business value.

2. Project Leadership & Cross-Functional Delivery

• Lead major data platform projects end-to-end, coordinating across engineering, analytics, IT and business teams.
• Serve as the technical lead for enterprise data migration programs.
• Follow a value-first delivery approach by prototyping early, reducing data debt and ensuring operational readiness.
• Create and maintain architecture documents, runbooks, technical documentation and project status updates.

3. Platform Architecture & Migration

• Design and implement enterprise Lakehouse, Data Lake and Data Warehouse architectures.
• Build and maintain enterprise Medallion Architecture (Bronze–Silver–Gold).
• Lead migration and decommissioning of SAP BW and on-premise SQL platforms, including historical data loads, validation and cutover.
• Define and enforce platform standards, reusable templates and engineering guardrails.
• Design integration patterns for Palantir Foundry and Aera within the Fabric/Databricks ecosystem.

4. Data Engineering & Pipeline Development

• Build enterprise-grade data pipelines using Microsoft Fabric Pipelines, Azure Data Factory and Databricks.
• Integrate SAP, CRM, ERP, IoT and SQL data sources using appropriate extraction and integration patterns.
• Implement real-time data pipelines using Eventstream and/or Event Hubs.
• Own transformation logic using Delta Live Tables, dbt and Fabric Notebooks.
• Reduce data duplication through virtualization and data-sharing patterns such as Shortcuts and Delta Sharing.
• Own pipeline monitoring, SLAs, alerting and incident response.

5. DataOps, Automation & Engineering Standards

• Define CI/CD standards using Azure DevOps and Databricks Asset Bundles.
• Implement automated data testing frameworks using DLT checks, Great Expectations and dbt tests.
• Continuously improve platform performance, cost efficiency, reliability and observability.
• Establish documentation standards and leverage AI-powered tools to streamline engineering documentation.

6. Data Products, Governance & Observability

• Design and deliver governed data products with clear ownership, SLAs and data contracts.
• Implement automated data quality controls across ingestion and transformation layers.
• Build and maintain enterprise data catalog and lineage using Microsoft Purview and Unity Catalog.
• Deploy data observability capabilities covering schema drift, data freshness, volume anomalies, lineage and cost monitoring.
• Define systems of record and enforce appropriate RBAC, RLS and data masking controls.

7. AI & Intelligence Platform Enablement

• Provide high-quality, governed and AI-ready data for ML, RAG and agentic AI workloads.
• Build data preparation pipelines supporting preprocessing, embeddings and retrieval use cases.
• Integrate Palantir Foundry and its ontology with Fabric/Databricks.
• Champion AI-powered engineering tools such as GitHub Copilot, Databricks AI Assistant and Azure AI.

Candidate Profile

Education

Required:

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, Data Engineering or a closely related technical discipline.

Preferred:

• Master's degree in Computer Science, Data Science or Information Management.
• Equivalent professional experience and relevant certifications may be considered in lieu of an advanced degree.

Experience

Required:

• 10+ years of hands-on experience in data engineering, data platforms or data architecture.
• 3–5 years at a senior/principal technical level, owning platform strategy and enterprise-scale technical direction.
• Proven experience leading complex, multi-workstream data platform projects end-to-end across engineering, analytics, IT and business teams.
• Experience owning and delivering an enterprise data platform strategy, including architecture decisions, technology selection and engineering standards.
• Hands-on experience with Microsoft Fabric, Databricks and/or Snowflake in enterprise Lakehouse environments.
• Strong experience designing enterprise data architectures covering Lakehouse, Data Lake, Data Warehouse and Medallion architectures.
• Proven experience executing on-premise-to-cloud migrations, including source mapping, historical data loads, validation and cutover.
• Experience establishing data governance foundations covering catalog, lineage, data quality, access controls and ownership models.
• Strong track record delivering multiple data integration patterns including ETL/ELT, replication, virtualization and streaming/event-based pipelines.
• Demonstrated experience with pipeline monitoring, data observability and incident management, including SLA ownership and root-cause analysis.
• Experience defining engineering standards and reusable frameworks for broader data engineering organizations.

Preferred:

• Experience working in manufacturing, chemicals or process-industry environments.
• Experience designing and delivering self-service or federated data products.
• Exposure to Palantir Foundry, including ontology, pipelines and AIP integration.
• Experience enabling data science/ML teams through feature pipelines, MLOps and model-scoring pipelines.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
• Microsoft Certified: Azure Data Engineer Associate (DP-203)