Data Platform Engineer

Job Responsibilities

Platform Design & Engineering

• Design and implement scalable, resilient, and secure data platforms (data lakes, lakehouse, data warehouses).
• Build and maintain distributed data systems that support batch, streaming, and real-time processing.
• Develop reusable frameworks and platform services to standardize data ingestion, processing, and access.

Data Pipeline Development

• Build and maintain robust ETL/ELT pipelines using modern orchestration tools.
• Ensure data quality, lineage, observability, and governance are embedded within pipelines.
• Optimize data workflows for performance, cost efficiency, and reliability.

Cloud & Infrastructure Management

• Deploy, manage, and optimize data platforms on cloud providers (Azure, AWS, GCP).
• Implement Infrastructure as Code (IaC) using tools like Terraform, ARM/Bicep, or CloudFormation.
• Monitor and manage platform performance, availability, and scalability.

Data Governance & Security

• Implement data governance frameworks, including cataloging, classification, and lineage tracking.
• Ensure compliance with data security and privacy standards (e.g., GDPR, PDPA).
• Manage access control, encryption, and auditing mechanisms.

DevOps & Automation

• Build CI/CD pipelines for data platform components.
• Automate deployments, monitoring, and alerting.
• Apply SRE principles to improve platform reliability and availability.

Collaboration & Enablement

• Partner with data engineers, data scientists, and business stakeholders to deliver data solutions.
• Provide platform best practices and guidelines to engineering teams.
• Support self-service data capabilities for analytics and AI use cases.

Requirement

Education & Experience

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering, platform engineering, or related roles.
• Experience with AI/ML data pipelines and feature stores.
• Knowledge of data security frameworks and zero-trust architecture.
• Certification in cloud platforms (e.g., Azure Data Engineer, AWS Certified Data Analytics).
• Familiarity with FinOps practices for data platform cost optimization.

Technical Skills

• Strong programming skills (Python, Java, Scala, or similar).
• Experience with distributed data processing frameworks (Spark, Flink, or equivalent).
• Proficiency in SQL and data modeling techniques.

Cloud & Data Technologies

• Hands-on experience with cloud data services:
• Azure: Data Factory, Synapse, Databricks, ADLS, HDInsight
• AWS: S3, Glue, Redshift, EMR
• GCP: BigQuery, Dataflow, Composer, Managed Spark, Hadoop
• Specialized Big Data: Snowflake, Databricks, Cloudera, Oracle Big Data Services
• Experience with modern data architectures (Lakehouse, Data Mesh, Data Fabric).

Data Pipeline & Orchestration

• Tools such as Airflow, Azure Data Factory, Prefect, or Dagster, Docket, Git.
• Experience with streaming platforms (Kafka, Redpanda Event Hubs, Kinesis).

DevOps & Infrastructure

• Familiarity with containerization (Docker) and orchestration (Kubernetes).
• Experience with CI/CD tools (Azure DevOps, GitHub Actions, Jenkins).
• Infrastructure as Code (Terraform preferred).

Data Governance, Security & Observability

• Experience with tools like Collibra, Purview, DataHub, Prometheus, Grafana, OpenLineage/ Apache Ranger
• Understanding data quality frameworks and monitoring tools.