Azure Databricks Lead

HCLTech is looking for a highly talented and self-motivated Azure Databricks Technical Lead to join it in advancing the technological world through innovation and creativity.

Job Title: Azure Databricks Technical Lead

Job ID: 156897

Position Type: Full-time (Hybrid)

Location: New York, NY

Role/Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS Gen2).
• Build and optimize batch and real-time data processing pipelines using PySpark and Spark SQL.
• Develop reusable, maintainable, and high-performance Python applications following OOP principles and advanced design patterns.
• Implement Delta Lake-based data solutions to support enterprise data lakehouse architectures.
• Design and optimize distributed data processing frameworks for large-scale datasets.
• Develop and maintain ETL/ELT processes ensuring data quality, reliability, and performance.
• Collaborate with business stakeholders, architects, and cross-functional teams to gather requirements and deliver scalable cloud data solutions.
• Monitor, troubleshoot, and optimize data pipelines and workloads running on Azure platforms.
• Support data governance, security, and compliance initiatives across enterprise data environments.
• Contribute to architecture discussions, code reviews, and best-practice implementation.

Minimum Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• Strong hands-on experience with:
• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Expert-level proficiency in Python programming.
• Strong understanding of Object-Oriented Programming (OOP) concepts and advanced Python design patterns.
• Extensive experience with:
• PySpark
• Spark SQL
• SQL
• Strong knowledge of:
• Apache Spark
• Delta Lake
• Data Lakehouse Architecture
• Distributed Computing Concepts
• Experience building scalable cloud-native data processing solutions.
• Strong analytical, troubleshooting, and performance optimization skills.
• Excellent communication and collaboration skills.

Level of Education

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, Data Analytics, or a related technical discipline.
• Equivalent professional experience may be considered.

Desired Qualifications

• Microsoft Azure Data Engineer Associate or related Azure certifications.
• Experience with CI/CD pipelines and DevOps practices for data engineering.
• Exposure to Azure Synapse Analytics, Azure Fabric, or other Azure analytics services.
• Experience implementing enterprise-scale data lakehouse solutions.
• Knowledge of data governance, data quality, and security frameworks.
• Experience working in Agile/Scrum environments.

Pay and Benefits

Pay Range Minimum: $127,000 per year

Pay Range Maximum: $180,000 per year

HCLTech is an equal opportunity employer, committed to providing equal employment opportunities to all applicants and employees regardless of race, religion, sex, color, age, national origin, pregnancy, sexual orientation, physical disability or genetic information, military or veteran status, or any other protected classification, in accordance with federal, state, and/or local law. Should any applicant have concerns about discrimination in the hiring process, they should provide a detailed report of those concerns to [email protected] for investigation.

A candidate's pay within the range will depend on their skills, experience, education, and other factors permitted by law. This role may also be eligible for performance-based bonuses subject to company policies. In addition, this role is eligible for the following benefits subject to company policies: medical, dental, vision, pharmacy, life, accidental death & dismemberment, and disability insurance; employee assistance program; 401(k) retirement plan; 10 days of paid time off per year (some positions are eligible for need-based leave with no designated number of leave days per year); and 10 paid holidays per year.

How You'll Grow

At HCLTech, we offer continuous opportunities for you to find your spark and grow with us. We want you to be happy and satisfied with your role and to really learn what type of work sparks your brilliance the best. Throughout your time with us, we offer transparent communication with senior-level employees, learning and career development programs at every level, and opportunities to experiment in different roles or even pivot industries. We believe that you should be in control of your career with unlimited opportunities to find the role that fits you best.