Senior Data Engineer

Data Engineer

Development, Cloud Engineering & DevOps

Cloud Platform: AWS (Compute, Storage, Networking, IAM and related cloud resources)

Data Engineering Skills: Python, PySpark, SQL, Prefect, Snowflake, Snowpark, dbt, airbyte, Databricks on AWS

DevOps Skills: CI/CD (AWS CodePipeline/CodeBuild/CodeDeploy), Terraform (IaC), Docker, Amazon CloudWatch, Secrets Management, Release & Environment Management

Must-Have Skills

• B.E./B.Tech degree in Computer Science, Engineering, or a related field, with 8-10 years of overall work experience.
• 5+ years of hands-on development experience building Cloud Data Platform Data Engineering solutions covering Data Ingestion, Data Quality Validations, Data Processing and Data Integration.
• Strong hands-on coding experience in Python, PySpark, and SparkSQL, with solid software engineering practices including testing, version control and code reviews.
• Hands-on experience in Airbyte, Snowflake and dbt.
• Hands-on experience with Prefect for workflow orchestration, including designing flows and tasks, scheduling, deployments, and parameterized runs.
• Ability to build reliable, observable Prefect workflows with retry logic, failure handling, and rerun/recovery support for production pipelines.
• Hands-on experience with Amazon S3-based data lakes, Databricks on AWS and other AWS-based data ecosystem services.
• Experience monitoring and troubleshooting orchestrated workflows via the Prefect UI/Cloud, including work pools, deployments and run history.
• Demonstrated willingness and ability to set up and own DevOps practices for the platforms you build (see DevOps Skills below).
• Proficiency in analytics use-case analysis, source system analysis, and data quality assessment.
• Experience coordinating/collaborating with on-shore and off-shore teams for solution delivery.
• Excellent communication and presentation skills.

DevOps Skills

• Design and build CI/CD pipelines using AWS CodePipeline, CodeBuild, and CodeDeploy (or equivalent tools such as GitHub Actions/Jenkins) to automate build, test, and release cycles.
• Provision and manage AWS cloud resources using Infrastructure as Code (Terraform), including version-controlled, reusable modules.
• Containerize applications and workflows with Docker; deploy and manage containers using Amazon ECS/EKS.
• Manage promotion of code and configuration across Development, Test, and Production environments with clear release and rollback strategies (blue-green/canary deployments).
• Implement secure secrets and configuration management using AWS Secrets Manager or Parameter Store, with least-privilege IAM policies.
• Set up monitoring, logging, and alerting using Amazon CloudWatch (metrics, dashboards, alarms) to maintain operational visibility into pipelines and workflows.
• Define and implement retry, recovery, and rerun strategies for failed jobs/workflows to ensure production reliability.
• Write automation scripts (Python/Bash) for deployment, operational tasks, and routine platform maintenance.
• Collaborate with data engineers, application developers, and cloud engineers to support end-to-end delivery, and troubleshoot production issues when required.

Nice-to-Have Skills

• Experience with Snowpark, and additional orchestration frameworks such as Amazon MWAA (Managed Airflow) or AWS Step Functions.
• Exposure to Docker and containerized deployments; familiarity with Amazon ECS/EKS (Kubernetes) is a plus.
• Experience with event-driven architectures (Amazon EventBridge, SQS, SNS, Lambda) and REST API integration across distributed services.
• Exposure to Data Management principles including Data Governance, Data Cataloging, and Master Data Management.
• Exposure to cloud-native MDM tooling.
• Exposure to BI analytics using Amazon QuickSight.