Data Engineer

Key Responsibilities

Technical Leadership & Ownership

• Own the end-to-end data engineering architecture for large-scale AWS data platforms
• Define and enforce data engineering standards, best practices, and governance frameworks
• Lead design reviews, code reviews, and technical decision-making across teams
• Act as the primary technical escalation point for complex data pipeline issues

ETL/ELT Design & Development

• Design, build, and optimize scalable ETL/ELT pipelines using:
• AWS Glue (Jobs, Workflows, Crawlers)
• PySpark / Spark SQL, Snowflake, SnowsQL
• Python-based data processing frameworks
• Implement incremental processing, CDC, and data partitioning strategies
• Develop reusable and modular data pipeline frameworks for enterprise use

Data Lake & Storage Management

• Design and manage data lake architecture on AWS (S3 + Apache Iceberg)
• Implement ACID-compliant data layers using Iceberg
• Optimize storage formats (Parquet, ORC) and data layouts for performance
• Define and enforce data lifecycle, retention, and archival policies

Performance Optimization & Cost Efficiency

• Tune Spark/Glue jobs for performance optimization (memory, partitioning, caching)
• Optimize workloads for cost efficiency in AWS (compute, storage, I/O)
• Monitor and improve pipeline SLAs, throughput, and latency metric

Data Governance & Quality

• Implement data quality frameworks, validations, and reconciliation checks
• Ensure compliance with data governance, lineage, and security standards
• Work with cataloging tools (AWS Glue Data Catalog, etc.) for metadata management

Integration & Orchestration

• Design and manage end-to-end orchestration workflows (Glue Workflows, Step Functions, Airflow if applicable)
• Integrate data across multiple sources (RDBMS, APIs, streaming platforms, files)
• Enable reliable, fault-tolerant, and restartable pipeline execution

Stakeholder Collaboration

• Partner with business, analytics, and AI teams to understand data requirements
• Collaborate with architects and DevOps teams for environment setup and automation
• Provide technical guidance to junior engineers and team members

Team Leadership & Mentoring

• Lead and mentor a team of data engineers
• Drive skill development in Spark, AWS, and modern data architectures
• Ensure adherence to Agile practices and timely delivery of milestones

Required Skills & Experience

Core Technical Skills

• Strong experience in AWS Data Engineering stack:
• AWS Glue, S3, Lambda, IAM, CloudWatch
• Advanced proficiency in:
• PySpark / Apache Spark
• Spark SQL
• Python
• Hands-on experience with Apache Iceberg / modern table formats
• Deep understanding of ETL/ELT design patterns and data pipelines

Data Engineering Expertise

• Experience with data lake and lakehouse architectures
• Strong knowledge of data modeling (star/snowflake schemas)
• Experience with batch and near real-time processing
• Familiarity with file formats (Parquet, ORC, Avro)

Performance & Optimization

• Proven experience in large-scale data processing (TB/PB scale)
• Strong expertise in query optimization, partitioning, and indexing strategies

DevOps & Automation

• Experience with CI/CD pipelines for data workflows
• Knowledge of infrastructure as code (CloudFormation/Terraform) is a plus
• Familiarity with version control (Git) and deployment strategies

Preferred Skills (Good to Have)

• Experience with data orchestration tools (Airflow, Step Functions)
• Exposure to streaming frameworks (Kafka, Kinesis)
• Knowledge of data security (encryption, masking, access control)
• Experience supporting AI/ML data pipelines
• Exposure to BI tools (Power BI, Tableau, Sigma)

Qualifications

• Bachelor’s/Master’s degree in Computer Science, Engineering, or related field
• 8–12+ years of experience in data engineering, with 3+ years in a technical leadership role