Data Engineer — Data Lakehouse
Indexed description
Domain: Regulated iGaming / wallet & ledger data. Audit-heavy: regulators, finance and analytics all consume the same tables. Millions of financial events per day, terabyte-plus scale.
What you'll be doing:
- Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution.
- Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events.
- Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
- Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion.
- Guarantee correctness: freshness SLAs, drift detection, reconciliation against the source wallet and ledger systems.
- Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails.
- Monitor ingestion health, data anomalies, and cloud storage/compute spend.
Must-have:
- 3 years' experience in hands-on delivery within that architecture — pipelines, ingestion, models, monitoring
- Lakehouse architecture — bronze/silver/gold layering, an open table format (Iceberg, Delta, or Hudi), schema evolution.
- Data layout & query optimization at TB+ scale — partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
- Cloud lakehouse/DWH in production — Snowflake, Databricks, or BigQuery.
- CDC & streaming ingestion — Kafka + Debezium or equivalent; late, duplicate and out-of-order events.
- Strong SQL and data modeling — enough relational grounding to reason about the OLTP systems you capture from. Critical for financial ledgers.
- Correctness — freshness SLAs, drift detection, reconciliation against source wallet/ledger systems.
- Governance — catalogs, lineage, row/column access control, PII masking, retention, GDPR deletion.
- Cloud object storage — S3 or GCS, plus storage tiering and archival.
- Python and an orchestrator — Airflow or Dagster, as tools.
Create a free Caio profile to unlock more results and save your role and location preferences.
Unlock free search