Data Engineer

We are seeking an experienced Data Engineer with strong expertise in PySpark, SQL, Azure Data Factory (ADF), Databricks, ETL, Python, Azure, and Snowflake. The role focuses on building reliable, governed, scalable, privacy-compliant, and cost-efficient data pipelines for an Azure-native third-party data enrichment platform.

Required Skills

• PySpark
• SQL
• Azure Data Factory (ADF)
• Databricks
• ETL
• Python
• Azure
• Snowflake
• Data Engineering
• Spark
• Architecture Design

Experience

• 5+ years of Data Engineering experience.
• 5+ years of hands-on experience with PySpark, SQL, and ADF.
• 5+ years of experience with Databricks, ETL, Python, and Azure.
• Strong experience developing and troubleshooting large-scale Spark data pipelines.
• Experience with Snowflake for analytics and data warehousing.
• Strong understanding of data governance, privacy, compliance, and access controls.

Key Responsibilities

• Build and enhance ingestion pipelines for large-scale batch and event-driven data processing.
• Integrate data from third-party enrichment vendors, digital platforms, Conversion API (CAPI) integrations, and rewards/promotions systems.
• Develop scalable ETL/ELT pipelines using PySpark, Databricks, Python, SQL, and ADF.
• Implement data validation, idempotency, replay/backfill, and deduplication strategies.
• Own monitoring, alerting, dashboarding, and operational readiness.
• Troubleshoot Spark failures using logs and perform root cause analysis.
• Diagnose Spark performance issues involving shuffle, skew, partitioning, and workload optimization.
• Ensure pipeline reliability and SLA adherence.
• Apply privacy, compliance, and governance requirements across pipelines and datasets.
• Support Unity Catalog, lineage, access controls, and PII/non-PII handling.
• Maintain documentation for tables, schemas, catalogs, and cluster usage.
• Optimize cluster sizing, compute/storage usage, and overall pipeline cost.
• Balance cost, quality, performance, and SLA requirements.

Good to Have

• Event-driven / streaming ingestion experience.
• Delta Live Tables (DLT).
• Databricks data engineering patterns.
• Configuration-driven data pipeline experience.

Desired Competencies

• Strong analytical and troubleshooting skills.
• Strong Spark performance optimization knowledge.
• Ability to troubleshoot production data pipeline issues independently.
• Strong understanding of data quality and reliability patterns.
• Knowledge of data governance, privacy, compliance, and lineage.
• Excellent communication and collaboration skills.
• Ability to work independently with minimal supervision.
• Strong ownership mindset.
• Ability to balance cost, quality, reliability, and SLA requirements.
• Experience working in Agile environments.

Key Deliverables

• Data Ingestion Pipeline Development
• ETL / ELT Pipeline Development
• PySpark & Spark Engineering
• Databricks Development
• Azure Data Factory Integration
• Snowflake Data Engineering
• Data Quality & Validation
• Data Governance & Compliance
• Data Lineage & Access Control
• Pipeline Monitoring & Reliability
• Spark Performance Optimization
• Cost Optimization
• Technical Documentation
• Architecture & Data Model Support

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.