AWS Data Platform

AWS Data Platform / Platform Engineering Lead

Location: Irvine, CA

Work Arrangement: Onsite – 5 Days/Week

Job Summary

We are seeking a highly experienced AWS Data Platform / Platform Engineering Lead to design, build, automate, and support enterprise-scale cloud data platforms and DevOps infrastructure.

The ideal candidate will have deep hands-on experience with AWS, Data Platform Engineering, Platform Engineering, Jenkins, CI/CD, Terraform, Infrastructure as Code (IaC), GitHub, SonarQube, ROC/D, and cloud-based DevOps.

This role will be responsible for building scalable and secure AWS data-platform infrastructure, developing automated CI/CD pipelines, implementing infrastructure automation, establishing code-quality controls, and improving platform reliability and operational excellence.

The candidate should be comfortable working across cloud infrastructure, data platforms, DevOps automation, CI/CD pipelines, infrastructure provisioning, deployment automation, and platform operations.

Key Responsibilities

AWS Cloud & Data Platform

• Architect, build, and manage enterprise AWS data platforms supporting data engineering, analytics, reporting, and business-critical workloads.
• Design scalable and highly available cloud-native data platform architectures.
• Build and manage AWS infrastructure supporting data ingestion, processing, storage, transformation, and analytics.
• Work extensively with AWS services such as S3, Glue, Redshift, Athena, Lambda, EMR, EKS, EC2, IAM, VPC, CloudWatch, KMS, and Secrets Manager.
• Develop secure and reusable cloud infrastructure patterns for data engineering and analytics teams.
• Implement AWS security, IAM, networking, encryption, access control, and governance.
• Support data-platform scalability, reliability, performance, availability, and cost optimization.
• Implement monitoring, logging, alerting, and observability for AWS data-platform environments.
• Collaborate with Data Engineering, Architecture, Security, Application, and Infrastructure teams.

Data Platform Engineering

• Build and support modern enterprise data-platform infrastructure.
• Support Data Lake / Data Lakehouse architectures and cloud-based data workloads.
• Enable data ingestion, batch processing, streaming, transformation, and analytical workloads.
• Support technologies such as Databricks, Spark/PySpark, dbt, Airflow, AWS Glue, and Redshift.
• Develop standardized infrastructure and deployment patterns for data engineering teams.
• Implement data-platform security, governance, monitoring, availability, and operational standards.
• Troubleshoot infrastructure and platform issues impacting data pipelines and analytics workloads.

Jenkins / CI/CD

• Design, develop, and maintain enterprise CI/CD workflows using Jenkins Pipelines.
• Build automated pipelines for source control, build, testing, quality validation, infrastructure provisioning, deployment, and release management.
• Install, configure, and maintain Jenkins plugins required for Git/GitHub repositories and enterprise CI/CD workflows.
• Configure SCM Polling, Git Webhooks, automated triggers, and pipeline orchestration.
• Develop reusable Jenkins pipeline frameworks and Shared Libraries.
• Integrate Jenkins with GitHub, Terraform, SonarQube, AWS, and other DevOps tools.
• Troubleshoot Jenkins pipeline failures, deployment issues, build failures, and integration problems.

SonarQube / Quality Gates

• Integrate SonarQube into Jenkins CI/CD pipelines.
• Configure and enforce SonarQube Quality Gates.
• Automate code-quality and security validation within CI/CD workflows.
• Ensure code and deployments meet defined enterprise quality standards.
• Monitor and resolve quality-gate failures before application or platform deployment.

Terraform / Infrastructure as Code

• Design and implement AWS infrastructure using Terraform.
• Develop reusable and standardized Terraform modules.
• Automate provisioning and configuration of AWS infrastructure and data-platform components.
• Integrate Terraform with Jenkins CI/CD pipelines.
• Implement automated Terraform plan, validation, approval, and deployment workflows.
• Manage infrastructure changes through Git-based version control.
• Establish enterprise standards for Infrastructure as Code, automation, security, and governance.

ROC/D & DevOps Automation

• Implement and support ROC/D within enterprise DevOps and CI/CD workflows.
• Integrate ROC/D with applicable source-control, pipeline, deployment, and infrastructure automation processes.
• Support automated release, deployment, and operational workflows involving ROC/D.
• Troubleshoot ROC/D-related deployment, pipeline, and platform issues.
• Work with engineering teams to standardize and automate release and operational processes.

Platform Engineering

• Establish enterprise Platform Engineering standards, automation frameworks, and reusable platform capabilities.
• Build self-service infrastructure and deployment capabilities for engineering and data teams.
• Standardize development, testing, and production deployment processes.
• Promote automation, GitOps, CI/CD, IaC, observability, security, and platform reliability.
• Reduce manual infrastructure and deployment activities through automation.
• Establish operational readiness, monitoring, incident management, and reliability practices.
• Continuously improve platform scalability, availability, security, and engineering productivity.

Required Skills

Mandatory Technical Skills

• AWS Cloud – Deep hands-on experience
• AWS Data Platform Engineering – Deep experience
• Platform Engineering
• Jenkins / Jenkins Pipelines
• CI/CD
• Terraform
• Infrastructure as Code (IaC)
• Git / GitHub
• SonarQube / Quality Gates
• ROC/D
• Cloud DevOps
• Infrastructure Automation
• Monitoring & Observability
• Python, Bash, or Shell scripting

AWS Skills

Strong hands-on experience with multiple AWS services:

S3 | AWS Glue | Redshift | Athena | Lambda | EMR | EKS | EC2 | IAM | VPC | CloudWatch | KMS | Secrets Manager

Candidate must understand how AWS services are integrated to create and operate enterprise data platforms.

Data Platform Skills

Strong understanding of:

• Data Lake / Data Lakehouse
• Enterprise Data Platforms
• Data ingestion and integration
• Batch and streaming data processing
• Data pipelines
• Data transformation
• Data storage and analytics
• Data platform security
• Data governance
• Data quality
• Platform monitoring and observability
• High availability and disaster recovery
• Data-platform performance and scalability

Databricks, Spark/PySpark, dbt, Airflow, AWS Glue, and Redshift experience is highly preferred.

Preferred Skills

• Databricks
• Apache Spark / PySpark
• dbt
• Apache Airflow
• AWS Glue
• Amazon Redshift
• Amazon EMR
• Kubernetes / Amazon EKS
• Docker
• GitOps
• Jenkins Shared Libraries
• Terraform Enterprise / Terraform Cloud
• Python
• Bash/Shell scripting
• Cloud security and governance
• Observability and monitoring
• Financial Services / Investment Management experience

Ideal Candidate Profile

The ideal candidate should be a hands-on Platform/Data Platform Engineer or Lead, not simply a traditional DevOps Engineer.

The candidate should demonstrate strong experience across:

AWS Cloud + Data Platform + Platform Engineering + Terraform/IaC + Jenkins CI/CD + GitHub + SonarQube + ROC/D + DevOps