Data Engineer

Senior Data Engineer

Fortune 50 Healthcare

Brooksource

Remote

Overview

Our Fortune 50 Healthcare client is seeking a Senior Data Engineer to support our mission of improving the health and well-being of our members. This role will focus on building scalable, secure, data centric solutions and compliant data platforms that power analytics, clinical insights, and business decision-making across the enterprise.

The ideal candidate will have strong experience with cloud-based data platforms, Databricks, PostgreSQL, and healthcare data, with a passion for delivering high-quality, trusted data solutions in a regulated environment.

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and scalable data pipeline solutions using Databricks (Spark) and cloud-native services
• Build and optimize ETL/ELT workflows for ingesting structured and unstructured healthcare data (claims, clinical, provider, and member data)
• Develop and maintain data models in PostgreSQL and enterprise data warehouses
• Support Lakehouse architecture leveraging Databricks, Delta Lake, and cloud storage
• Improve performance, reliability, and cost-efficiency of data platforms

Healthcare Data & Compliance

• Work with healthcare datasets, including producer/agent, broker, commission, and distribution data, ensuring proper ingestion, normalization, and optimization for analytics and reporting
• Ensure compliance with HIPAA, HITECH, and enterprise data governance policies
• Implement data security, encryption, masking, and access controls
• Maintain data lineage, auditability, and regulatory reporting readiness

Advanced Data Processing

• Build real-time and batch pipelines for analytics and operational use cases
• Develop data transformations using PySpark and SQL within Databricks
• Leverage PostgreSQL for transactional and analytical workloads where applicable
• Integrate data from APIs, third-party vendors, and internal systems

Collaboration & Stakeholder Engagement

• Partner with business stakeholders to support data-driven initiatives and member acquisition strategies
• Translate insurance distribution, agent/producer, and marketing requirements into scalable, high-quality data solutions
• Support downstream consumers, including Power BI, marketing analytics teams, and operational reporting stakeholders, by delivering curated, analytics-ready datasets

Technical Leadership

• Lead design and architecture discussions for enterprise data solutions
• Establish and enforce best practices in data engineering, testing, and CI/CD
• Contribute to enterprise data strategy and platform modernization

AI & Advanced Analytics (Databricks Genie)

• Leverage Databricks Genie (AI/BI capabilities) to enable natural language querying and democratize data access for business stakeholders
• Design and optimize semantic layers and governed datasets that power Genie-driven insights with trusted, high-quality data
• Collaborate with stakeholders to translate business questions into AI-assisted analytics workflows using Databricks
• Ensure AI outputs are accurate, explainable, and compliant with healthcare data governance and HIPAA requirements
• Leverage large language models (LLMs), including Anthropic Claude, to enhance data exploration, automate insight generation, and support conversational analytics use cases
• Integrate Genie capabilities with Delta Lake and curated data models to support near real-time insights and decision-making
• Partner with data scientists and analytics teams to enhance AI-driven use cases, including producer performance insights, marketing attribution, and member engagement analysis

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• 5–8+ years of experience in data engineering
• Strong programming in Python (PySpark) and advanced SQL
• Hands-on experience with:
• Databricks (core requirement)
• PostgreSQL
• Distributed data processing frameworks (Apache Spark)
• Experience with cloud platforms (Azure preferred; AWS acceptable)
• Proficiency in building and maintaining ETL/ELT pipelines
• Strong understanding of data modeling and warehousing concepts

Preferred Qualifications

• Experience in healthcare or insurance industry (payer experience strongly preferred)
• Familiarity with healthcare standards (e.g., FHIR, HL7)
• Experience with:
• Delta Lake / Lakehouse architecture
• Orchestration tools (Airflow, Azure Data Factory)
• Streaming (Kafka, Event Hubs)
• Knowledge of DevOps and CI/CD pipelines (Azure DevOps, GitHub Actions)
• Experience supporting machine learning pipelines

Key Skills & Competencies

• Deep understanding of data pipelines at scale
• Strong experience with Databricks ecosystem and Spark optimization
• Expertise in PostgreSQL performance tuning and schema design
• Strong attention to data quality, governance, and compliance
• Excellent communication skills, especially with non-technical stakeholders
• Ability to work in a highly regulated healthcare environment

Typical Technology Stack

• Data Platform: Databricks, Delta Lake
• Database: PostgreSQL, Snowflake (optional)
• Cloud: Azure, Google, AWS
• Languages: Python, SQL
• Orchestration: Airflow, Azure Data Factory
• Visualization: Power BI
• Version Control: Git

KPIs / Success Metrics

• Reliability and performance of Databricks pipelines
• Data quality and compliance adherence (HIPAA standards)
• Time-to-delivery for new data products
• Query performance improvements in PostgreSQL and data warehouse systems
• Stakeholder adoption and satisfaction