Data Engineering Tech Lead

Every data team says they care about quality. Most of them are still firefighting the same pipeline failures every Monday morning.

We’re looking for a Tech Lead who ends that cycle. Someone who defines how data is built, structured, and governed — and makes those standards stick. Not by managing people, but by setting a technical bar so clear that the right path becomes the easy path.

This is a senior Individual Contributor role. The Team Lead manages delivery and people. You own technical direction.

Why This Matters

Deriv’s mission is Trading for Anyone, Anywhere, Anytime. Millions of traders, around the clock, across regulatory regimes. Data powers everything: trading decisions, compliance pipelines, product analytics, AI/ML systems. At this volume, a silent pipeline failure isn’t a minor inconvenience. It’s a compliance risk, a broken model, a wrong number on a dashboard someone is making decisions from.

Real money, real regulations, real consequences. The data engineering team exists to make sure the infrastructure underneath all of this is accurate, observable, and governed. You’ll be the technical authority who ensures it stays that way.

Why Deriv

We’re in production, not planning.

• Dozens of fraud detection models running continuously against live transactions.
• AI resolving 65%+ of customer enquiries with genuine judgment, not decision trees.
• Finance automation processing real invoices, reconciliations, and procurement workflows.
• Open-source Spec-to-PR transforming product specs into full implementations.
• 400+ internal users on our workflow orchestration platform.

None of this works without data pipelines that are reliable, well-modelled, and governed. You’ll own the technical quality of the infrastructure layer that makes all of it possible.

We share openly. Deriv is where we write about what we’re shipping, what breaks, and what we figured out the hard way. We’re also working to open-source projects, so your contributions may go beyond Deriv.

What You’ll Do

Technical Authority

• Define, document, and drive adoption of technical standards across the data engineering team — pipeline patterns, modelling conventions, integration approaches, code review expectations.
• Make architectural decisions for pipelines, modelling layers, and integration patterns. Review and align the team’s technical choices against those decisions.
• Contribute through reference designs, design reviews, and critical implementation decisions. You work alongside the team, not above it.
• Act as the senior technical reviewer for production-critical systems and design changes.

Production Reliability

• Own production reliability for data pipelines. That means incident prevention, post-incident reviews, and systematic reduction of recurring failures.
• Build observability into every pipeline: freshness checks, completeness validation, schema drift detection, lineage tracking. If it’s not observable, it’s not production-ready.
• Drive durable fixes from root cause. Patches buy time; your solutions are permanent.

Scalable Architecture

• Design dimensional models, medallion layers, and integration patterns that hold as volume and complexity grow — not just for today’s requirements, but for the next order of magnitude.
• Enforce data contracts between producers and consumers. SLAs, SLOs, and schema agreements are engineering deliverables, not handshake agreements.
• Automate scheduling, testing, deployment, and quality gates. Manual processes are debt; you pay them down systematically.

Team Technical Velocity

• Raise the team’s technical ceiling through design reviews, pairing, and code review standards that teach, not just gatekeep.
• Make the right path the easy path: documented patterns, reusable components, clear conventions. When a new engineer joins, they should be building correctly within their first week.
• Influence through architecture, standards, and mentoring. Your leverage is in what the whole team ships, not just what you build alone.

Governance by Design

• Embed PII handling, access controls, and auditability into the pipeline itself. Governance is code, not a checklist someone runs quarterly.
• Automate data quality checks and anomaly detection so compliance is continuous, not periodic.
• Treat data contracts and SLAs as first-class engineering deliverables with the same rigour as application code.

Who You Are

You’ve done this before.

• 8+ years in data engineering, with a proven track record in technical leadership or a principal engineer capacity. You’ve shipped production-grade data infrastructure, not just prototyped it. You’ve seen what breaks at scale and built systems that don’t.

You think in architecture, not just queries.

• Strong experience with data modelling — Kimball star schemas, Data Vault, dimensional modelling, medallion architecture. You make practical trade-offs between purity and pragmatism, and you can explain why.

You know your stack cold.

• Deep experience with a cloud data stack — GCP/BigQuery/Airflow/Python or equivalent. Strong hands-on experience with dbt, Dataform, or similar transformation and semantic layer tools. You’ve optimised BigQuery for cost and performance, and you understand partitioning, clustering, and orchestration reliability at the operational level.

You ship with discipline.

• CI/CD for data pipelines is how you work, not an aspiration. Version control, automated testing, deployment pipelines. You’ve built and enforced data contracts, SLAs, and quality frameworks at production scale.

You use AI as a force multiplier.

• AI coding assistants are part of your daily workflow — for design quality, code review, and engineering throughput. You’re pragmatic about what AI accelerates and what still requires human judgment.

You lead through craft.

• Excellent communication skills. You convey technical decisions clearly to non-technical stakeholders. You mentor senior and mid-level engineers through working code and design reviews, not presentations. You balance speed, reliability, cost, and governance in every architectural decision.

You build, review, and ship alongside the team.

• This is a hands-on role. You don’t just draw architecture diagrams and walk away. You write code, review code, and own production outcomes.

What Sets You Apart

• Experience designing and operating real-time or event-driven pipelines — Kafka, Pub/Sub, or equivalent.
• Exposure to low-code integration tools like Fivetran or RudderStack.
• Experience with pipeline observability tooling — lineage, alerting, anomaly detection.
• Familiarity with open table formats — Apache Iceberg, Delta Lake, or similar.
• Background in financial services, fintech, or regulated environments where data accuracy is non-negotiable.
• A track record of defining and documenting technical standards that a team actually adopted and maintained.

Tech Stack

• Cloud: GCP, BigQuery
• Orchestration: Apache Airflow
• Transformation: dbt / Dataform
• Languages: Python, SQL
• Streaming: Kafka, Pub/Sub
• CI/CD: Version control, automated testing, deployment pipelines
• AI Assistants: Daily use as part of engineering workflow

The Honest Reality

This is demanding work. You’ll own technical direction for pipelines where a wrong number means a compliance breach, a broken model, or a trading decision based on bad data. You’ll make architectural calls with incomplete information and live with the consequences. You’ll push standards on engineers who are under pressure to ship fast. Some of those conversations will be uncomfortable.

But you’ll shape how an entire team builds data infrastructure. You’ll see your patterns and standards adopted across production systems that handle real financial transactions. And you’ll work in an environment where technical quality is valued, not just tolerated.

If you want a role where someone else defines the standards and you just execute, this isn’t it. If you want technical ownership with the latitude to define how a high-performing team builds, it might be.