Observability and Evaluation Engineer

An Observability and Evaluation Engineer (often specialized in AI/LLM systems) designs the tracking, tracing, and testing frameworks that monitor production behavior, measure output quality, and catch regressions in complex software or agentic AI workflows. [1, 2, 3, 4]

Core Responsibilities

• Build Observability Pipelines: Instrument applications and LLM/agent pipelines using telemetry tools (like OpenTelemetry, Arize, Galileo, or LangSmith) to capture execution traces, run logs, and latency metrics. [1]
• Design Evaluation Frameworks: Create offline and online evaluation suites to benchmark accuracy, groundedness, toxicity, tool-use correctness, and reasoning-chain validity. []
• Implement Regression & Drift Testing: Build automated test harnesses and continuous evaluation gates to detect model degradation, data drift, or output anomalies before releases reach production. [1]
• Root-Cause Analysis: Investigate execution traces and multi-turn interaction failures to diagnose erratic system behaviors, API misparameters, or bottlenecks. []
• Optimize Performance & Cost: Monitor and balance operational telemetry relating to token consumption, execution speed, and infrastructure costs. [1]

Key Skills & Requirements

• Programming: Strong proficiency in Python or Go for building custom evaluation scripts and hooking into telemetry SDKs.
• Tools & Stacks: Experience with OpenTelemetry, Prometheus, Grafana, or dedicated AI observability platforms (e.g., Arize Phoenix, LangChain/LangSmith, Galileo, Braintrust).
• Testing Methodologies: Designing adversarial prompts, red-teaming protocols, and golden datasets for LLM/system validation.
• System Design: Understanding of distributed microservices or LLM orchestration layers (LangChain, LlamaIndex, AutoGen). [1, 2, 3]

If you are tailoring this for a specific application, would you like me to focus this job description more heavily on traditional cloud/microservices observability or Generative AI / LLM agent evaluation?