Senior Azure Databricks Platform Engineer

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A ServiceNow-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A “Great Place To Work® Certified™” company, Prodapt employs over 6,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan. Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

We are seeking a highly skilled Senior Azure Databricks Platform Engineer to lead the deployment, migration, optimization, and governance of enterprise-scale Databricks platforms on Azure. The ideal candidate brings deep expertise in Databricks architecture, Azure cloud services, data platform administration, and production support, with a strong focus on performance, scalability, reliability, and automation.

KEY RESPONSIBILITIES

• Prepare and configure the Databricks Serverless environment for enterprise teams, ensuring all required libraries, dependencies, and platform components are available and maintained.
• Design, deploy, and manage Databricks job clusters and workflows.
• Collaborate with Databricks and enterprise stakeholders to enable and optimize AI/ML capabilities across the platform.
• Migrate existing Databricks jobs, workflows, and data processing pipelines to modernized environments.
• Troubleshoot and resolve Azure networking, proxy, firewall, authentication, and connectivity issues affecting Databricks and related applications.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Enhance operational monitoring and observability by building solutions that track job execution status, start/end times, duration, failures, and overall health.
• Maintain and enhance the DataSphere health check framework and associated automation processes.
• Implement and support Unity Catalog, access controls, governance policies, and security best practices.
• Investigate and resolve production incidents, failed jobs, and data pipeline issues.
• Create and manage Databricks Genie Spaces for business domains as part of migration and modernization initiatives.

REQUIRED QUALIFICATIONS

• 10+ years in Data Engineering / Cloud Data Platforms, with strong Azure Databricks administration and migration expertise
• Bachelor's or Master's degree in Computer Engineering, or in a related field.
• Strong hands-on experience with Azure Databricks, including:
• Workflows, jobs, and cluster management
• Serverless compute
• Performance tuning and optimization
• Proven experience with Microsoft Azure Cloud, including:
• Virtual networking
• Firewalls and proxy configurations
• Authentication and authorization
• Application deployments
• Strong programming skills in Python and SQL.
• Experience with ETL/ELT development, migration projects, and modern data engineering practices, including direct Databricks and Snowflake platform experience.
• Expertise in production support, monitoring, alerting, incident management, and job reliability improvements.
• Experience with Unity Catalog, data governance, access management, and data quality frameworks.

PREFERRED QUALIFICATIONS

• Experience with Databricks AI/ML capabilities and generative AI initiatives, including LLM-powered applications and AI-driven workflows.
• Experience building operational dashboards and health-check automation frameworks.
• Knowledge of SSO, OAuth, Microsoft Entra ID, and identity management solutions.
• Experience developing platform monitoring, observability, and operational support portals.
• Familiarity with enterprise-scale cloud data platforms and governance frameworks.

DESIRED SKILLS

Databricks | Azure Cloud | Python | SQL | Snowflake | Unity Catalog | ETL/ELT | Data Governance | Monitoring & Observability | React | TypeScript | Azure Web Apps | AI/LLM | Entra ID | Production Support