SC Cleared Data Engineer

AWS Data Engineer (Must hold current SC clearance)

Overview:
We are seeking a highly skilled AWS Data Engineer to join a cutting-edge technology consultancy delivering solutions across the UK and Europe. This role is ideal for someone with strong experience in building scalable data pipelines and working within secure, cloud-based environments.

You will collaborate with data scientists, analysts, and software engineers to deliver efficient and reliable data solutions that support analytics, reporting, and AI/ML initiatives.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS services such as Glue, Lambda, Step Functions, and Kinesis.

• Work with structured and unstructured data from multiple sources, ensuring efficient ingestion, transformation, and storage.

• Build and optimise data lake and warehouse solutions using S3, Redshift, Athena, and Lake Formation.

• Implement data governance, security, and compliance best practices, including IAM, encryption, and access controls.

• Monitor and optimise performance of data workflows using CloudWatch and other AWS-native tools.

• Automate data processes using Python, PySpark, SQL, and AWS SDKs.

• Collaborate with cross-functional teams to support analytics, BI, and AI/ML use cases.

• Maintain and enhance CI/CD pipelines for data infrastructure using Terraform, CloudFormation, or CDK.

• Troubleshoot and resolve data integration, performance, and reliability issues.

Required Skills & Experience:

• 5+ years of experience in data engineering, with a strong focus on AWS.

• Strong proficiency in Python, PySpark, SQL, and AWS Glue.

• Hands-on experience with AWS data services including Redshift, Athena, Glue, EMR, and Kinesis.

• Solid understanding of data modelling, warehousing, and schema design.

• Experience with streaming and event-driven architectures (e.g. Kafka or Kinesis).

• Expertise in Infrastructure as Code (Terraform, CloudFormation, or CDK).

• Familiarity with DevOps practices and CI/CD pipelines.

• Strong problem-solving skills and experience working in Agile environments.

Desirable:

• AWS certifications (Data Analytics Specialty or Solutions Architect Associate).

• Experience with Airflow for orchestration.

• Exposure to big data frameworks such as Spark, Hadoop, or Presto.

• Experience supporting AI/ML data pipelines.