Azure Databricks + AI Engineer

Azure Databricks + AI Engineer

Location: Chicago, IL

Job Summary

We are looking for an experienced Azure Databricks + AI Engineer to build scalable data, AI/ML, and Generative AI solutions on the Azure platform. The ideal candidate should have strong hands-on expertise in Azure Databricks, PySpark, Python, SQL, and GenAI.

Key Responsibilities

• Design and develop data and AI solutions using Azure Databricks and Azure services.
• Build scalable pipelines using PySpark, Python, SQL, Delta Lake, and Lakehouse architecture.
• Develop AI/ML and Generative AI solutions using LLMs, RAG, embeddings, and vector search.
• Work with Databricks ML, MLflow, Mosaic AI, and model deployment workflows.
• Integrate Databricks with ADLS Gen2, Azure Data Factory, Azure Synapse, Azure OpenAI, and other Azure services.
• Implement MLOps, CI/CD, monitoring, security, and governance.
• Optimize Spark jobs, data pipelines, and AI workloads for performance and cost.
• Collaborate with data scientists, engineers, architects, and business teams.

Required Skills

• 6+ years of experience in Data Engineering / AI / ML.
• Strong hands-on Azure Databricks, PySpark, Python, and SQL.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Experience with Azure Data Factory and ADLS Gen2.
• Hands-on experience with GenAI, LLMs, RAG, embeddings, and vector databases.
• Experience with Azure OpenAI, MLflow, or Databricks ML.
• Understanding of MLOps, CI/CD, cloud security, and data governance.