Databricks Data Engineer

Databricks Field Data Engineer - Reference Solution Architect

DATABRICKS FIELD DATA ENGINEER / REFERENCE SOLUTION ARCHITECT

OVERVIEW

We are seeking a highly skilled Databricks Field Data Engineer / Reference Solution Architect to join our Engineering department. As a senior member of the team, you will be responsible for designing and implementing scalable data solutions using Databricks and other cloud-based technologies. You will work closely with cross-functional teams to develop and deploy data-intensive applications that drive business growth.

KEY RESPONSIBILITIES

• Design and implement data pipelines using Apache Spark, PySpark, and SQL on Databricks, leveraging Delta Lake and Medallion Architecture for optimal performance.
• Collaborate with engineering teams to develop and deploy CI/CD pipelines on cloud platforms such as AWS, Azure, and GCP.
• Develop reference architectures and solutions using Unity Catalog, ensuring seamless data management and governance.
• Work with stakeholders to identify and implement use cases for Generative AI, RAG, LLM, and Agentic AI, driving innovation and business value.
• Provide technical leadership and guidance on data engineering best practices, ensuring high-quality solutions that meet business requirements.
• Develop and deliver technical training and workshops to internal teams, enhancing their skills and knowledge of Databricks and related technologies.

REQUIREMENTS

• Proficiency in Apache Spark, PySpark, and SQL, with experience in designing and implementing scalable data pipelines.
• Strong knowledge of Databricks, Delta Lake, and Medallion Architecture, with experience in developing high-performance data solutions.
• Experience with cloud platforms such as AWS, Azure, and GCP, with a deep understanding of CI/CD pipelines and deployment strategies.
• Familiarity with Unity Catalog and data governance principles, with experience in developing and implementing data management solutions.
• Strong understanding of Generative AI, RAG, LLM, and Agentic AI, with experience in identifying and implementing use cases that drive business value.
• Excellent communication and collaboration skills, with experience in working with cross-functional teams and stakeholders.

NICE TO HAVE

• Experience with agile development methodologies and version control systems.
• Knowledge of data science and machine learning principles, with experience in developing and deploying models using Databricks and related technologies.
• Familiarity with DevOps practices and tools, with experience in developing and implementing automated testing and deployment scripts.