Senior Data Engineer

The Senior Data Platform Engineer is a senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities. Reporting to the Sr. Director of Data Platform Transformations & Operations, this role combines hands-on engineering, solution design, and operational ownership to deliver scalable, secure, and reliable solutions. The engineer serves as a technical lead on major initiatives while remaining accountable for production support and operational excellence.

The role supports platform transformation and steady-state operations, with approximately 70% focused on development of new capabilities and 30% on operations, support, and platform reliability. Primary areas include Azure Databricks, cloud data engineering, platform modernization, automation, governance, and continuous improvement.

Responsibilities

• Design, develop, and maintain scalable data pipelines, ingestion frameworks, transformation processes, and reusable data products using Azure Databricks, PySpark, SQL, and Delta Lake.

• Implement Bronze, Silver, and Gold architecture patterns supporting enterprise reporting, analytics, AI, and self-service data consumption.

• Build reusable frameworks, utilities, and platform components that improve engineering productivity, quality, consistency, and deployment speed.

• Develop and support batch, near-real-time, and streaming integration solutions, including modernization of legacy warehouse and ETL workloads.

• Serve as technical lead for complex initiatives; develop solution designs, lead technical reviews, recommend tools and approaches, and guide work through production implementation.

• Partner with data architects and platform leaders to ensure solutions are scalable, secure, governed, cost-conscious, and operationally supportable.

• Mentor engineers and promote standards for coding, testing, documentation, performance, and production readiness.

• Implement data quality, validation, reconciliation, monitoring, metadata, and lineage capabilities; support RBAC and enterprise security controls.

• Build and maintain CI/CD, automated testing, deployment, and release processes using Azure DevOps and Git-based practices.

• Contribute to platform observability, alerting, operational dashboards, health metrics, performance tuning, and cost optimization.

• Participate in production support, incident response, pager, and on-call rotations; troubleshoot issues, lead root cause analysis, and implement durable remediation.

• Create and maintain operational runbooks, support procedures, technical documentation, and knowledge-sharing assets.

• Collaborate with architecture, governance, security, analytics, application, and business teams to translate requirements into scalable platform solutions.

• Support technical discovery, estimation, roadmap planning, delivery execution, and evaluation of emerging cloud, data, and AI capabilities.

Job Requirements

Education

• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Science, or a related field.

• Master's degree preferred.

Experience

• 10+ years of experience in data engineering, data warehousing, or data platform engineering.

• 5+ years of experience designing or implementing cloud-based data warehouse, data lake, or data platform solutions.

• 3+ years of hands-on Azure Databricks experience in enterprise production environments.

• Experience with ETL/ELT development, data modeling, large-scale integration, and cloud platform modernization.

• Experience leading technical implementations, solution design, design reviews, and production releases.

• Experience supporting production environments, incident response, root cause analysis, and operational support processes.

• Experience in regulated, financial services, banking, or audit-sensitive environments preferred.

Skills & Competencies

• Advanced proficiency in SQL, Python, PySpark, and Spark performance optimization.

• Strong experience with Azure Databricks, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.

• Strong understanding of lakehouse, Medallion Architecture, dimensional modeling, data warehousing, and analytics-oriented data structures.

• Experience with Azure DevOps, Git, CI/CD, automated testing, and deployment practices; Databricks Asset Bundles experience preferred.

• Experience implementing data quality, validation, reconciliation, monitoring, metadata, lineage, governance, and security controls.

• Strong technical leadership, solution design, analytical, troubleshooting, and problem-solving skills.

• Ability to independently lead complex work from design through production support and balance innovation with reliability, security, cost, and supportability.

• Excellent collaboration, mentoring, documentation, and communication skills across technical and business teams.

• Experience with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development preferred.

• Databricks Data Engineer certification at the Associate or Professional level preferred.