Data Engineering Architect - Digital Operations

Data Engineering Architect – Digital Operations

Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. Fractal is building a world where individual choices, freedom, and diversity are the greatest assets. An ecosystem where human imagination is at the heart of every decision. Where no possibility is written off, only challenged to get better. We believe that a true Fractalite is the one who empowers imagination with intelligence. Fractal has been featured as a Great Place to Work by The Economic Times in partnership with the Great Place to Work® Institute and recognized as a ‘Cool Vendor’ and a ‘Vendor to Watch’ by Gartner.

Please visit Fractal | Intelligence for Imagination for more information about Fractal.

Note: This position is not eligible for Immigration Sponsorship at this time.

Position Overview:

We are seeking an experienced Data Engineering Architect to join our Digital Operations team, providing operational excellence for enterprise-scale Azure Data and Analytics platforms. This role is primarily focused on ensuring platform stability, reliability, performance, and continuous improvement of cloud-based data solutions through proactive monitoring, incident management, root cause analysis, and stakeholder engagement.

The ideal candidate will have a strong background in Azure data engineering technologies, production support, and operational governance, with the ability to work across engineering, business, and support teams to maintain critical data services and meet business SLAs. This role requires a Digital Operations mindset, focusing on operational reliability, service management, incident response, platform observability, and continuous service improvement.

Experience in the Consumer-Packaged Goods (CPG), Retail, or a comparable high-velocity data domain is strongly valued. Familiarity with domain-specific data patterns such as supply chain analytics, demand forecasting, point-of-sale pipelines, or omnichannel reporting will be a meaningful differentiator.

Key Responsibilities:

Production Support & Operations

• Provide L2/L3 support for Azure Data and Analytics platforms and services.
• Manage and resolve production incidents, service requests, and critical issues within defined SLAs.
• Monitor data pipelines, workflows, system health, and platform performance using Azure monitoring tools.
• Analyze failures, data quality issues, and performance bottlenecks to ensure minimal business disruption.
• Participate in on-call support and major incident management activities as required.

Incident & Problem Management

• Perform detailed Root Cause Analysis (RCA) for recurring incidents and platform issues.
• Implement corrective and preventive actions to improve platform stability.
• Collaborate with engineering teams to deploy permanent fixes and operational enhancements.
• Drive continuous improvement initiatives to reduce incident volumes and improve operational efficiency.

Platform Reliability & Governance

• Ensure availability, scalability, and reliability of Azure-based data platforms.
• Establish and enforce operational best practices, support procedures, and runbooks.
• Maintain operational documentation, incident reports, and knowledge repositories.

Technical Responsibilities:

Azure Data Factory (ADF)

• Triage and resolve failed or stuck ADF pipeline runs, including debugging activity-level errors, dependency failures, and trigger misconfiguration.
• Analyze integration runtime (IR) performance and connectivity issues between on-premises and cloud environments.
• Optimize pipeline execution by reviewing parallelism settings, retry policies, and data movement activities.
• Maintain and update linked services, datasets, and parameterized pipeline templates to support evolving operational needs.

Databricks

• Monitor and troubleshoot Databricks job clusters, workflows, and notebook executions in production environments.
• Diagnose and resolve Spark job failures, including out-of-memory errors, executor loss, shuffle partitioning, and skewed data joins.
• Optimize Databricks notebooks and jobs for performance, including Delta Lake read/write optimization, Z-ordering, VACUUM operations, and partition tuning.
• Manage Databricks cluster configurations (autoscaling, spot instances, cluster policies) to balance cost and performance.
• Automate recurring operational tasks using Databricks CLI, REST APIs, and Databricks Workflows (Jobs API).

Azure Data Lake Storage (ADLS)

• Investigate and resolve data ingestion failures, file format inconsistencies (Parquet, Delta, CSV), and storage permission errors.
• Monitor ADLS Gen2 storage metrics — capacity usage, transaction rates, latency — and raise alerts for anomalies.
• Manage hierarchical namespace configurations, access control lists (ACLs), and Azure RBAC assignments.
• Coordinate data lifecycle management policies to control storage costs and data retention.

Azure Monitoring & Observability

• Build and maintain Azure Monitor dashboards, alerts, and Log Analytics (KQL) queries for proactive platform health monitoring.
• Configure diagnostic settings and integrate platform logs with centralized monitoring solutions (Azure Monitor, Application Insights, or equivalent).
• Define and track SLA/SLO metrics for critical data pipelines and platform services.
• Develop automated alerting for pipeline failures, SLA breaches, data quality anomalies, and infrastructure health events.

Data Quality & Validation

• Implement and monitor data quality checks across ingestion, transformation, and serving layers.
• Investigate data reconciliation issues, duplicate records, schema drifts, and late-arriving data scenarios.
• Collaborate with engineering teams to embed automated data quality frameworks in production pipelines.

Required Qualifications:

• 7+ years of experience in data engineering or analytics engineering roles, with at least 3 years in a production support or platform operations capacity.
• Strong communication and stakeholder management skills; ability to translate technical issues into clear business impact and provide timely incident updates to non-technical audiences.
• Demonstrated experience working within CPG, Retail, or a comparable domain with complex, high-volume data pipelines and tight SLA environments.
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field (or equivalent practical experience).

Preferred Qualifications:

• Experience with CI/CD pipelines for data platform deployments using Azure DevOps or GitHub Actions, including branching strategies, release gates, and environment-specific configurations.
• Familiarity with Delta Live Tables (DLT) troubleshooting pipeline graph failures, data quality expectations, and incremental processing.
• Hands-on experience with Unity Catalog governance tasks, including access control, data lineage review, and schema management.
• Experience with infrastructure-as-code (IaC) practices using Terraform or ARM/Bicep templates for data platform provisioning.
• Advanced KQL skills for complex diagnostic queries and custom alerting rules beyond standard dashboards.

Pay:

The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Fractal, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is: $125,000 - $170,000. In addition, you may be eligible for a discretionary bonus for the current performance period.

Benefits:

As a full-time employee of the company or as an hourly employee working more than 30 hours per week, you will be eligible to participate in the health, dental, vision, life insurance, and disability plans in accordance with the plan documents, which may be amended from time to time. You will be eligible for benefits on the first day of employment with the Company. In addition, you are eligible to participate in the Company 401(k) Plan after 30 days of employment, in accordance with the applicable plan terms. The Company provides for 11 paid holidays and 12 weeks of Parental Leave. We also follow a “free time” PTO policy, allowing you the flexibility to take the time needed for either sick time or vacation.

Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation