Data Engineer

We are looking for a hands-on Data Engineer to help build and manage our data platform for reporting, analytics, and future data science use cases.

The role will involve working with SQL, Python, PySpark, AWS, ETL/ELT pipelines, data warehouses, and BI/reporting tools. Our architecture may use technologies such as ClickHouse, Redshift, AWS Glue, Airflow, Step Functions, Lambda, S3, and CDC-based replication tools based on scale, cost, and operational needs.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT data pipelines from multiple databases, applications, and external systems.
• Build raw, cleaned, and business-ready data layers to support reporting, analytics, and future data science use cases.
• Write efficient SQL, Python, and PySpark jobs for data ingestion, transformation, validation, and processing.
• Implement workflow orchestration using Apache Airflow, AWS Glue, Step Functions, or similar tools.
• Work with data warehouses such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.
• Support cross-service reporting as the architecture moves towards independent microservice databases.
• Build reusable reporting tables, aggregates, summaries, and basic data marts.
• Monitor, troubleshoot, and optimize data pipelines, warehouse queries, and processing jobs.
• Implement data quality checks for freshness, completeness, consistency, duplicates, and reconciliation.
• Support BI/reporting needs through tools such as Power BI, Metabase, Superset, Redash, or similar.
• Apply data governance, access control, security, and PII-handling best practices.
• Collaborate with engineering, DevOps, product, business, finance, risk, and support teams.

Required Skills

• Strong expertise in SQL for joins, aggregations, window functions, query optimization, and analytical reporting.
• Hands-on experience with Python for data processing, automation, validation, and scripting.
• Working experience with PySpark / Apache Spark for processing large datasets.
• Good understanding of ETL/ELT pipelines, data warehousing, and data modelling concepts.
• Experience with workflow orchestration using Apache Airflow, AWS Step Functions, AWS Glue, or similar tools.
• Experience with AWS data services such as S3, Glue, Lambda, Step Functions, Redshift, DMS, CloudWatch, or similar.
• Experience with any data warehouse such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.
• Understanding of relational databases, preferably PostgreSQL.
• Ability to debug data mismatches, failed pipelines, slow queries, and data quality issues.
• Exposure to BI tools such as Power BI, Metabase, Superset, Redash, or similar.

Good ownership, problem-solving, communication, and collaboration skills.

Skills:- Data engineering, ClickHouse, Amazon Web Services (AWS) and PySpark