Senior BigQuery Data Engineer

The Role: We seek a highly skilled Senior BigQuery Data Engineer with strong Python expertise to design and deliver high-performance data solutions within OneMIS. This pivotal role supports complex reporting functions and leads the migration of on-premise data platforms to Google Cloud Platform (GCP). You will manage large-scale financial datasets, ensuring secure, compliant, and low-latency data solutions for analytics and decision-making. This role demands technical mastery, problem-solving acumen, and a deep understanding of data best practices in the financial sector.

Key Responsibilities:

Data Platform Architecture & Modernization:

• Design, develop, and maintain scalable BigQuery data warehouse solutions (architecture, schema, data modeling).
• Lead cloud-native data warehouse design and implementation to replace on-premise infrastructure.
• Optimize BigQuery datasets, tables, and views for performance, cost, and governance through partitioning, clustering, query tuning, and workload management.

Data Pipeline Engineering:

• Build robust data processing pipelines and workflows feeding BigQuery from diverse sources.
• Develop and maintain efficient ETL/ELT processes for large-volume financial data ingestion, transformation, and loading.
• Implement comprehensive data quality checks, validation, and reconciliation processes.

GCP Ecosystem & Python Development:

• Utilize and optimize core GCP services: BigQuery, Cloud Composer (Apache Airflow) for workflow orchestration, Dataflow (Apache Beam) for batch/streaming, and Cloud Storage.
• Develop, test, and deploy high-quality Python code for data processing, automation, API integrations, and custom solutions, adhering to best practices (Git, testing, documentation).
• Explore and integrate other relevant GCP technologies (e.g., GKE, Cloud Functions, Pub/Sub) and implement security, access control, and compliance.

Collaboration & Agile Practices:

• Collaborate with product owners, analysts, and engineering teams to align solutions with business requirements.
• Actively participate in agile ceremonies and foster team collaboration.

Business Intelligence & Financial Expertise:

• Support data needs for BI tools (Qlik Sense, Tableau, Looker).
• Apply strong understanding of banking/financial sector data, regulations, and reporting for compliance.

Skills & Experience:

Required:

• 5+ years of data engineering experience.
• Strong SQL knowledge, preferably on Google Cloud Platform (GCP).
• Expert-level Python programming skills for data manipulation, scripting, and application development.
• Proven experience with Cloud Computing, demonstrating practical application of cloud services.
• Direct hands-on experience with core GCP services: BigQuery, Cloud Composer (Apache Airflow) for DAG development/orchestration, Dataflow (Apache Beam) for batch/streaming pipelines, and Cloud Storage.
• Good understanding of data warehousing concepts, data flows, and data feeds.
• Experience with Git-based source control, particularly Bitbucket.
• Experience working in an agile development environment with familiar agile ceremonies.
• Solid understanding of the banking and financial sector, including data types and regulatory landscapes.
• Ability to work collaboratively in a dynamic environment.

Nice to Have:

• Experience with other GCP services (GKE, Cloud Functions, Pub/Sub, Dataproc).
• Familiarity with CI/CD pipelines and DevOps practices.
• Experience with Infrastructure as Code (Terraform).
• Knowledge of data visualization tools (Qlik Sense, Tableau, Looker).
• Understanding of data governance, data quality, and metadata management.
• Experience with real-time data processing.

Education:

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related quantitative field.