System Reliability Engineer (AI Solution) (T00025620)

Job summary:

AI Platform / System Reliability Engineer to run the AI platform and the stabilization period for deployed AI/GenAI solutions, owning the reliability, availability, cost, and security of AI workloads. This is a critical role because the AI Centre of Excellence fully owns stabilization: you will operate monitoring and on-call, harden workloads, manage capacity and cost, and build the runbooks and knowledge base that allow each solution to be safely transferred to the receiving unit with a proven SLO track record. The ideal candidate combines solid platform and operations skills with a calm, evidence-led approach to incidents and a security-first mindset.

Job description:

• Operates monitoring, alerting, and on-call for AI/GenAI workloads — infrastructure, model endpoints, data pipelines, and guardrails.
• Hardens AI workloads: secrets, key rotation, network segmentation, endpoint protection, and dependency security.
• Operates capacity, autoscaling, and cost controls across GPU/CPU, token budgets, and quota management.
• Builds and maintains stabilization-period runbooks and the L2/L3 knowledge base for owned services, and participates in dry-runs.
• Takes on-call shifts and performs L1/L2 incident response, documenting post-incident actions for AI-specific failure modes such as hallucination spikes, prompt injection, drift, and cost runaway.
• Owns service-level objectives (SLOs), service-level indicators (SLIs), and error budgets for assigned AI services, keeping availability, latency, cost, and quality within target.
• Supports stabilization hand-overs by training the receiving unit on assigned runbooks.
• Identifies opportunities to improve monitoring, automation, and incident-response practices, and contributes to the platform reference runtime and golden paths.
• Structures incident triage, identifies dependencies, and escalates ambiguity early, resolving known AI-workload issues effectively.
• Reduces repeat incidents within scope and helps execute clean stabilization hand-overs within the agreed SLA.

Qualifications:

• Bachelor's degree in Computer Science, Computer/Network Engineering, Information Technology, or a related field (or equivalent practical experience).
• 1–6 years of SRE, DevOps, platform, or infrastructure-operations experience.
• Solid Linux and networking fundamentals, and comfort with at least one cloud platform — Microsoft Azure preferred.
• Hands-on experience with monitoring/alerting and log analytics (Azure Monitor, Prometheus/Grafana, Application Insights, or similar).
• Scripting and automation ability (Python, Bash, or PowerShell) and basic infrastructure-as-code (Terraform/Bicep).
• Familiarity with containers and Kubernetes (AKS) operations: deployments, scaling, and log/metric inspection.
• Understanding of secrets management, key rotation, and basic security hardening.
• Foundational understanding of LLM-ops — model endpoints, token-cost basics — and AI failure modes such as drift, prompt injection, and cost runaway.
• Willingness to participate in an on-call rotation during stabilization.
• Experience in banking, financial services, or another regulated environment is an advantage; comfortable working in an Agile / Scrum environment.