Senior/Lead Data Engineer - PySpark/Azure Databricks

Description

Senior Data Engineer(AI) / Lead Data Engineer(AI) for Leading MNC

A leading MNC is looking for highly skilled Data Engineering professionals with expertise in Databricks, PySpark, Spark Optimization, CI/CD, and AI/ML data workflows.

Location : Gurgaon

Employment Type : Permanent

Experience : 5+ Years

Key Skills

• Python & PySpark
• Apache Spark & Spark Performance Optimization
• Databricks, Delta Lake & Distributed Data Processing
• GitLab CI/CD Pipelines & Automation
• Databricks REST APIs & SQL Optimization
• Scalable Data Platform Architecture
• Vector Search & Vector-Space Architectures
• AI/ML Workflows, LLMOps & RAG Pipelines
• Data Governance, Metadata & Data Quality

Key Responsibilities

• Design and build scalable enterprise data platforms using Databricks, Spark, Delta Lake, and PySpark
• Develop and optimize large-scale distributed data pipelines and processing applications
• Perform Spark tuning, query optimization, and troubleshooting for performance improvements
• Build and maintain enterprise-grade CI/CD pipelines using GitLab and automation tools
• Work on Databricks automation, REST APIs, and deployment workflows
• Support advanced AI/ML initiatives including LLMOps, RAG pipelines, vector search, and model evaluation
• Implement data governance, security, metadata management, and quality standards
• Mentor junior engineers and drive engineering best practices across teams

Please share the following details along with your resume :

• Current CTC :
• Expected CTC :
• Experience :
• Notice Period :

(ref:hirist.tech)