Databricks architect

Hybird 2-3 days week

Architect the lakehouse: Design and own scalable, secure Databricks Lakehouse architecture on Azure (Delta Lake, Unity Catalog, medallion bronze/silver/gold, ADLS Gen2) aligned to enterprise standards.

•

Stay hands-on: Personally build and review PySpark / Spark SQL pipelines, Delta Live Tables, notebooks, and orchestration — setting the engineering bar, not just directing it.

•

Lead legacy migration: Drive conversion of complex legacy ETL (DataStage) workloads to Databricks/PySpark and ADF, including patterns, accelerators, and reusable frameworks for code conversion and validation.

•

Own performance & cost: Optimize cluster configuration, job performance, partitioning, and cost; establish FinOps and right-sizing practices on Databricks.

•

Embed governance: Implement data governance, lineage, quality, and access control through Unity Catalog and Purview; ensure security, privacy, and compliance by design.

•

Enable analytics & AI: Design Gold-layer semantic models and feature pipelines that serve BI (Power BI), advanced analytics, and ML/GenAI use cases (MLflow, Azure ML).

•

Lead the squad: Provide technical leadership and mentoring to data engineers; define best practices, coding standards, CI/CD (Azure DevOps), and review processes.

•

Partner with the client: Work closely with the client’s VP (Data & AI), AVP (Data Platforms & Integration), platform architects, and business stakeholders to translate requirements into delivery roadmaps and measurable outcomes.

Required Qualifications (Must-Have)

•

12+ years in data engineering / data platform architecture, with 4+ years of deep, hands-on Databricks delivery.