AI Systems Support Engineer

The AI Systems Support Engineer is responsible for the reliability, availability, and resilience of DAMAC AI's GPU cluster infrastructure — ensuring Blackwell-based DGX SuperPOD and HGX systems achieve and exceed defined SLA and uptime commitments. This role combines traditional site reliability engineering with deep expertise in GPU infrastructure failure modes, disaster recovery, business continuity, and the operational resilience of mission-critical AI compute environments.

Experience Requirements

7+ years of experience in Site Reliability Engineering, infrastructure operations, or HPC systems administration in data center or cloud environments

Deep understanding of GPU infrastructure failure modes — GPU hardware faults, NVLink errors, InfiniBand fabric partitions, and storage path failures

Proven experience designing and executing Disaster Recovery plans and Business Continuity exercises for mission-critical infrastructure

Strong experience with SLO/SLI frameworks, error budget management, and reliability engineering practices

Hands-on experience with observability platforms: Prometheus, Grafana, NVIDIA DCGM, ELK/Loki, PagerDuty/OpsGenie

Experience leading P1/P2 incident response and facilitating blameless postmortems in high-availability environments

Proficiency in Python, Bash, or Go for automation, self-healing tooling, and observability integrations

Familiarity with NVIDIA DGX SuperPOD architecture, NVLink/NVSwitch fabric design, and NVIDIA DCGM health monitoring

Incident Management & Root Cause Analysis

Lead P1 and P2 incident response for GPU infrastructure events — detection, triage, escalation, containment, recovery, and post-incident review

Drive root cause analysis (RCA) and blameless post-mortem process for all P1 incidents — ensuring findings are documented and remediation actions tracked to closure

Maintain MTTR targets: ≤2 hours for P1 GPU infrastructure events, ≤4 hours for P2 events

Build and maintain on-call runbooks, escalation playbooks, and incident response procedures for AI infrastructure teams

Operate the incident management workflow in ServiceNow or PagerDuty — ensuring all incidents are tracked, communicated, and closed with full documentation

Establish a weekly reliability review meeting to assess open incidents, SLO burn, and upcoming maintenance risk