Databricks Architect

Databricks Architect

Location: New Jersey (NJ)

Job Type: Full-Time

Experience: 8+ years

Work Arrangement: Hybrid / On-site as required

Job Summary

We are seeking an experienced Databricks Architect to design, implement, and optimize modern data platforms using Databricks, Apache Spark, and cloud technologies. The ideal candidate will have strong experience in data architecture, data engineering, cloud platforms, data lakehouse architecture, and enterprise data integration.

The architect will work closely with business stakeholders, data engineers, data scientists, and application teams to build scalable, secure, and high-performance data solutions.

Key Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse architectures.
• Develop scalable data pipelines using Databricks, Apache Spark, PySpark, and SQL.
• Design solutions using Delta Lake, Delta Live Tables (DLT/Lakeflow), Unity Catalog, and Workflows.
• Develop batch and real-time/streaming data processing solutions.
• Integrate Databricks with enterprise data sources, APIs, databases, and cloud storage.
• Design and implement data ingestion, transformation, and data quality frameworks.
• Establish data governance, security, access controls, lineage, and compliance using Unity Catalog.
• Optimize Spark jobs, SQL queries, clusters, workloads, and overall platform performance.
• Define best practices for Databricks development, deployment, CI/CD, and DevOps.
• Collaborate with cloud and infrastructure teams on platform architecture and scalability.
• Lead architecture discussions, technical design sessions, and proof-of-concept initiatives.
• Provide technical leadership and mentorship to data engineering teams.
• Ensure solutions meet enterprise standards for security, reliability, scalability, and performance.

Required Skills

• 8+ years of experience in data engineering, data architecture, or cloud data platforms.
• 4+ years of hands-on experience with Databricks.
• Strong knowledge of Apache Spark, PySpark, Python, and SQL.
• Strong understanding of Delta Lake and Lakehouse architecture.
• Experience with Unity Catalog, Databricks Workflows, DLT/Lakeflow, and Databricks SQL.
• Experience designing data platforms on at least one major cloud:
• Azure
• AWS
• Google Cloud Platform (GCP)
• Strong experience with ETL/ELT pipelines and enterprise data integration.
• Experience with relational and NoSQL databases.
• Knowledge of data modeling, dimensional modeling, and data warehousing concepts.
• Experience with CI/CD, Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, or similar tools.
• Strong understanding of data security, governance, monitoring, and performance optimization.
• Excellent communication and stakeholder-management skills.

Preferred Qualifications

• Databricks Certified Data Engineer / Data Architect certification.
• Experience with Azure Data Factory, AWS Glue, Amazon S3, Azure Data Lake Storage (ADLS), or equivalent services.
• Experience with Kafka or other streaming technologies.
• Experience implementing real-time analytics solutions.
• Knowledge of MLflow and MLOps.
• Experience with Power BI, Tableau, or other BI platforms.
• Experience with enterprise data governance and regulatory requirements.
• Prior experience working in financial services, healthcare, insurance, pharmaceutical, or other regulated industries.

Education

Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

Key Technologies

Databricks | Apache Spark | PySpark | Python | SQL | Delta Lake | Unity Catalog | Lakehouse | DLT/Lakeflow | Databricks SQL | Azure/AWS/GCP | Kafka | Terraform | Git | CI/CD | Data Governance | Data Engineering