Senior/Principal Engineer (m/f/x) for Evaluations Generative AI

Dynatrace Senior/Principal Engineer (m/f/x) for Evaluations Generative AI Reposted 17 Hours AgoSaved Remote or Hybrid Boston, MA, USA 74K-112K Annually Senior level 74K-112K Annually Senior levelArtificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • AutomationDesign and build evaluation and simulation systems for generative AI agents using Dynatrace observability data. Create large-scale simulation pipelines, define metrics/datasets/judging strategies, build developer CLIs, generate adversarial scenarios, and measure agent tool-use and failure modes. Prototype, run feedback cycles, set technical strategy, and mentor engineers.Top Skills: Agent FrameworksApmAWSAzureCi/CdCli ToolingDynatraceEvaluation FrameworksGolden DatasetsGCPLlm-As-A-JudgeLlmsObservabilityPairwise ComparisonsPrompt EngineeringTrace Ingestion