Data Engineer

Job Title: Databricks Engineer

Location: Melbourne, VIC (Hybrid)

Experience: 5–8+ Years

About the Role

Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.

Key Responsibilities

Databricks Platform Development

• Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
• Build and optimise Apache Spark jobs for large-scale data processing and transformation
• Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
• Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
• Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
• Develop notebooks and reusable libraries using Python, Scala, or SQL

Data Engineering & Pipelines

• Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
• Work with structured, semi-structured, and unstructured data sources
• Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
• Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
• Implement data quality checks, validation, and observability across pipelines
• Manage Unity Catalog for data governance, lineage, and access control

Machine Learning & Analytics

• Build and manage MLflow experiments, model tracking, and model registry
• Support data scientists in operationalising ML models on Databricks
• Develop feature engineering pipelines for ML workloads
• Implement Databricks AutoML and experiment management best practices

Cloud & DevOps

• Deploy and manage Databricks workspaces on AWS, Azure, or GCP
• Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
• Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
• Implement GitOps practices for notebook and pipeline version control
• Monitor and optimise Databricks workloads for performance and cost efficiency

Governance & Security

• Implement Unity Catalog for data governance, metadata management, and access control
• Ensure data lineage, traceability, and compliance across all data assets
• Apply row-level and column-level security across Delta Lake tables
• Document data models, pipeline architectures, and operational runbooks

Required Skills & Experience

• 5–8+ years of experience in data engineering or a related role
• Strong hands-on experience with Databricks Lakehouse Platform (mandatory)
• Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
• Strong proficiency in Python — data engineering and pipeline development (mandatory)
• Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
• Experience with Delta Live Tables (DLT) and Databricks Workflows
• Strong SQL skills — complex querying and data transformation
• Experience with Unity Catalog — data governance and access control
• Experience with MLflow — experiment tracking and model registry
• Hands-on experience with cloud platforms — AWS, Azure, or GCP
• Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
• Experience with Terraform for infrastructure as code
• Experience working in Agile / Scrum delivery environments