Data Engineer (GCP)

Responsibilities:

• Architect, build, and maintain production-grade ELT/ETL data pipelines that stream and batch high-volume, complex datasets into Google BigQuery.
• Manage, monitor, and upgrade the Apache Airflow or Cloud Composer platform.
• Write clean, modular, and maintainable Python DAGs, implement custom operators and hooks, and optimize cluster performance.
• Maintain and optimize core Google Cloud Platform data services, including BigQuery, Cloud Storage, Cloud Functions or Cloud Run, Pub/Sub, and IAM.
• Implement advanced BigQuery performance tuning, including partitioning, clustering, slot management, and query cost control.
• Develop high-performance Python scripts, API connectors, and automation tools for data extraction, schema validation, and third-party data integrations.
• Enforce data quality frameworks, monitoring, and automated alerting across ingestion pipelines to minimize data loss and latency.
• Maintain infrastructure-as-code and CI/CD deployment pipelines for data workloads using Git.
• Assist in architectural reviews and mentor junior engineers on data warehousing best practices.

Requirements:

• Software Engineering Experience: 4–7+ years of professional software development experience, with at least 3+ years focused heavily on data warehousing, cloud data pipelines, and data platform operations.
• Advanced Python Mastery: Production-grade Python development skills for data processing, custom pipeline tooling, and API integrations.
• Airflow Platform Operations: Hands-on experience with Apache Airflow (Cloud Composer preferred), including DAG architecture, troubleshooting, environment configuration, and platform version upgrades.
• GCP Ecosystem Expertise: Deep experience with the Google Cloud analytics suite—specifically BigQuery, Cloud Storage (GCS), Cloud Composer, Pub/Sub, and Cloud Functions/Run.
• BigQuery Optimization: Advanced understanding of BigQuery architecture, DDL/DML, partitioning, clustering, slot allocation, and query cost-control strategies.
• DevOps & Version Control: Proficient with Git, CI/CD pipelines, and modern software development workflows.

Nice to have:

• GCP Certification: Google Cloud Certified Professional Data Engineer (highly preferred).
• High-Volume Data Experience: Experience working with multi-terabyte or billion-row datasets, unstructured web-scraped data, or streaming architectures.
• Security & Compliance: Familiarity with government compliance frameworks, IAM policy design, and data encryption standards.