Data Engineer – Cary, NC

Job Description:

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.

• Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala

• Implement and support Data Lake / Lakehouse architectures

• Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.

• Expertise in handling Data migration using API’s.

• Apply data quality checks and support data governance basics (metadata, lineage, access controls)

• Optimize Spark jobs for performance, reliability, and cost

• Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Required Skills & Experience:

• 8+ years of experience in data engineering in azure using Azure Databricks.

• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.

• Strong Data Engineering experience in Big Data (Cloudera)

• Hands-on expertise in SQL, Python and PySpark or Scala

• Proven experience with batch & real-time ingestion

• Experience with Kafka and big data processing

• Added good to have experience with:

• Informatica PowerCenter

• Informatica IDMC or equivalent cloud data integration tools

• SQL Server, Oracle, Azure SQL

• Cosmos DB

• Unix/Linux

• Maestro (or similar enterprise schedulers)

• Data tools such as Precisely or equivalent

Added Advantage

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)

• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD

• Practical usage of Copilot for data engineering and development productivity

• Exposure to Generative AI / GenAI use cases in data platforms and analytics

Role Descriptions: Microsoft Azure (ADB/ADF)

Essential Skills: Microsoft Azure (ADB/ADF)

Desirable Skills:

Keyword:

Skills: Digital : Microsoft Azure

Experience Required: 6-8