Senior Data Engineer/Data Engineer Team Lead

Responsibilities:

• Own and deliver end-to-end Data Migration and Data Engineering projects, independently managing all phases from Source Assessment and Data Profiling to Production deployment and post-production support.
• Assess and analyze source systems, including Source Assessment, Data Profiling, and Data Mapping, to identify data structures, dependencies, and potential migration risks.
• Design Data Architecture and Data Models aligned with business requirements, scalability, performance, and cost considerations.
• Develop and maintain Batch and Real-time/Streaming Data Pipelines using appropriate technologies and cloud platforms.
• Execute data migration across On-premise, AWS, and GCP environments, ensuring data integrity, security, and minimal disruption.
• Perform Data Quality, Validation, and Reconciliation to ensure data accuracy, completeness, and traceability.
• Plan and execute Cutover and Rollback strategies to minimize downtime and migration risks.
• Deploy, monitor, troubleshoot, and provide post-production support for data pipelines and platforms.
• Take full ownership of assigned projects with minimal dependency on other teams.
• Design solution architecture, data models, and migration strategies covering Full Load, Incremental/CDC, batch, and real-time/streaming workloads.
• Own the full delivery process single-handedly: source profiling, data mapping, schema conversion, pipeline development, orchestration, testing, data quality/reconciliation, performance tuning, deployment, monitoring, cutover, rollback, and hypercare.
• Lead and execute large-scale data migration projects (e.g., on-premise to GCP/AWS, database replatforming to AlloyDB/BigQuery) with data validation and zero/minimal-downtime cutover.
• Build and operate data pipelines and platforms on GCP and AWS using AlloyDB/PostgreSQL, BigQuery, Cloud Storage/S3, Airflow/Cloud Composer, Spark, Flink, Kafka, Pub/Sub, and Kubernetes/GKE.
• Design lakehouse/data warehouse solutions using Parquet and Apache Iceberg, including partitioning, clustering, compaction, schema evolution, and data lifecycle management.
• Establish standards for CI/CD, Infrastructure as Code, observability, logging/alerting, data lineage, metadata, IAM, encryption, privacy, and cost optimization; analyze production incidents and root causes.
• Mentor and guide data engineers on best practices; lead technical design and code reviews (Team Lead level).
• Collaborate with data analysts, data scientists, application teams, and business stakeholders to deliver reliable data products.

Technical Leadership & Solution Design:

• Translate business requirements into scalable and practical technical solutions independently.
• Evaluate and select appropriate technologies and tools based on cost, timeline, performance, scalability, and business requirements.
• Serve as a Technical Lead, establishing best practices, development standards, CI/CD processes, and Data Governance frameworks.
• Provide technical guidance and consultation to Data Engineers and continuously improve the team's technical capabilities.
• Conduct Code Reviews, Technical Coaching, knowledge sharing, and technical documentation to ensure consistent engineering standards.

Key Objectives:

• Successfully deliver large-scale Data Migration projects with high data accuracy, completeness, traceability, and minimal downtime.
• Design and build scalable Data Platforms / Lakehouse architectures on GCP and AWS to support growing data volumes, processing requirements, and user demand.
• Build highly reliable, observable, and cost-efficient Data Pipelines, with effective monitoring, alerting, and cost optimization.
• Establish engineering standards and continuously improve the team's capabilities through Code Reviews, Technical Coaching, documentation, and best practices.
• Collaborate closely with Data Analysts, Data Scientists, Application Teams, and Business Users to deliver trusted, high-quality, and business-ready data.

Qualifications:

• Strong proficiency in SQL (window functions, CTE, query optimization, execution plans) and Python (Scala/Java is a plus).
• Hands-on experience with relational/distributed databases, data warehouses, data lake/lakehouse, and messaging/stream processing systems.
• Deep understanding of data migration patterns: CDC, idempotency, exactly-once/at-least-once semantics, data quality, reconciliation, and lineage.
• Able to design and deliver production systems end-to-end independently, including architecture documents, data mapping, test evidence, runbooks, and operational handover.
• Solid understanding of data consistency, fault tolerance, disaster recovery, security, privacy, and performance/cost optimization at scale.
• Able to act as Technical Lead / Team Lead: break down work, estimate effort, review architecture/design/code, coach the team, and coordinate across teams.
• Strong leadership, ownership, and communication skills with business, application, infrastructure, security teams, and vendors; able to make technical decisions under time, quality, risk, and cost constraints.
• Experience with Debezium, Dataflow/Apache Beam, Dataproc, Datastream, dbt, Terraform, Docker, Delta Lake/Hudi, Snowflake/Redshift, or legacy/on-premise systems is a plus.

Technical Skills:

• Cloud Platform: Google Cloud Platform (หลัก), AWS (รอง)
• Data Warehouse / Database: BigQuery, AlloyDB, Cloud SQL, PostgreSQL, Oracle, SQL Server, Redshift
• Data Processing: Apache Spark (PySpark), Apache Beam / Dataflow, Apache Flink, Dataproc
• Orchestration: Apache Airflow / Cloud Composer (Dagster หรือ Prefect เป็น plus)
• Streaming & Messaging: Apache Kafka, Google Pub/Sub, Kinesis / MSK
• Storage & Table Format: Cloud Storage, Amazon S3, Parquet, Avro, ORC, Apache Iceberg, Delta Lake, Apache Hudi
• Container & Infrastructure: Docker, Kubernetes (GKE / EKS), Terraform, Helm
• Programming: Python, SQL (Scala / Java พิจารณาเป็นพิเศษ)
• DevOps & Monitoring: Git, CI/CD Pipeline, Cloud Logging / Monitoring, Prometheus, Grafana

Soft Skills & Working Style:

• Demonstrates strong end-to-end ownership, proactively managing tasks from initiation to completion and effectively handling ad-hoc issues.
• Strong systematic problem-solving and analytical skills, with the ability to identify root causes of complex data-related issues.
• Communicates effectively with both technical teams and business users, translating complex technical concepts into clear, easy-to-understand language.
• Maintains strong documentation discipline, including Technical Design, Data Dictionary, Runbook, and Handover Documents.
• Focuses on code and data quality through Code Reviews, Unit Testing, and Data Validation.
• Able to manage multiple projects simultaneously, prioritize tasks effectively, and accurately estimate timelines.
• At the Team Lead level, capable of planning team workloads, delegating tasks, coaching team members, and providing technical guidance.
• Highly adaptable and eager to learn new technologies through self-directed learning.

Data Security & Compliance:

• Good understanding of Data Security principles, including Data Masking, Encryption at Rest/In Transit, Identity and Access Management (IAM), and Row-Level/Column-Level Security.
• Good understanding of PDPA requirements and the ability to design data pipelines in compliance with data protection regulations.

General Qualifications:

• Strong ability to read and understand technical documentation in English.
• Able to communicate effectively in English when working with technical teams and international vendors.