Lead Data Engineer
Indexed description
General Responsibilities
The Lead Data Engineer is responsible for designing, developing, and optimizing data infrastructure to ensure reliable pipelines, scalable architecture, and high-quality data for analytics and business needs. The role includes leading a team of data engineers, collaborating with cross-functional stakeholders, and implementing best practices in data transformation, validation, governance, and cloud-based solutions.
Duties & Responsibilities
- Lead and mentor a team of data engineers, fostering collaboration, innovation, and continuous improvement.
- Define and enforce data engineering standards, coding practices, and architectural guidelines.
- Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable solutions.
- Design and maintain high-performance ETL/ELT pipelines and workflows.
- Build and optimize data lake, data warehouse, and streaming data solutions.
- Ensure systems support structured, semi-structured, and unstructured data sources.
- Develop data transformation and preparation processes to deliver clean, reliable, analytics-ready datasets.
- Implement validation, anomaly detection, and automated quality checks to ensure accuracy and consistency.
- Define and execute data remediation strategies to resolve issues with minimal business disruption.
- Implement and manage cloud-based data platforms (AWS, Azure, or GCP).
- Apply modern frameworks (e.g., Spark, Kafka, Airflow, DBT) for efficient data processing.
- Oversee data modeling, schema design, and database optimization (SQL/NoSQL).
- Ensure accuracy, consistency, and availability of data across platforms.
- Implement governance, lineage, validation frameworks, and security best practices.
- Monitor, troubleshoot, and optimize pipelines to meet SLAs and performance benchmarks.
- Stay current with emerging data engineering and analytics technologies.
- Recommend improvements to systems, tools, and processes to enhance scalability, reliability, and quality.
- Other job-related activities that may be assigned from time to time
- Total Data Engineering experience - 7- 8yrs and above
- Skilled in Python (atleast 4/5 rating), SQL (atleast 3/5 rating)
- Working or has experience working on modern data cloud platforms (Databricks/Snowflake)
- With exposure to cloud services (preferably Azure or AWS)
- Wiling to work on a hybrid work set up in BGC Taguig
Create a free Caio profile to unlock more results and save your role and location preferences.
Unlock free search