ML Infrastructure Engineer
Indexed description
About the Role
We're a seed-stage enterprise AI infrastructure company building the context layer that makes AI agents reliable, accurate, and secure for critical business operations — including highly regulated industries like insurance, banking, asset management, and healthcare. Our platform automatically constructs a governed, real-time domain model across all enterprise data, enabling AI agents to make confident, auditable decisions in production.
As an ML Infrastructure Engineer, you'll own the systems that keep our agents running reliably and fast at scale. This is a hands-on production engineering role — focused on real-world impact, not research. You'll design, build, and scale our inference and model-serving infrastructure as concurrency and customer demands grow.
What You'll Do
Own inference and model-serving infrastructure end to end — from initial design through production deployment and ongoing scaling.
Build and scale systems that enable AI agents to run reliably and efficiently under high and increasing concurrency.
Identify and resolve infrastructure bottlenecks in collaboration with ML and platform engineering teams.
Optimize systems for latency, throughput, and reliability across cloud-hosted production environments.
Drive observability, monitoring, and debugging practices across our production ML stack.
What We're Looking For
Dealbreakers — all required:
5+ years of hands-on experience building and operating machine learning inference systems, model-serving platforms, or ML infrastructure in production environments.
Demonstrated experience designing and scaling inference-serving infrastructure using tools such as TensorFlow Serving, TorchServe, Triton, KServe, or equivalent custom systems.
Proven ability to optimize production ML systems for latency, throughput, and reliability at scale.
Required skills:
Experience with containerization and orchestration (Docker, Kubernetes) for deploying and scaling ML workloads.
Background in distributed systems that handle high concurrency and dynamic resource allocation under load.
Proficiency with monitoring and observability tooling — e.g., Prometheus, Grafana, ELK stack, distributed tracing.
Experience deploying and managing ML systems on cloud platforms (AWS, GCP, or Azure).
Proficiency in at least one systems or backend language: Python, Go, Rust, C++, or Java.
Nice to have:
Experience with knowledge graphs, semantic search, or graph databases (e.g., Neo4j, Amazon Neptune, or similar).
Background in real-time inference or low-latency serving requirements.
Familiarity with agentic AI systems, autonomous agents, or multi-step reasoning pipelines.
Experience with enterprise data infrastructure, data pipelines, or data integration platforms.
Location & Work Arrangement
This is a full-time, on-site role based in San Mateo, CA. On-site collaboration is an important part of how this small, fast-moving team operates. Visa sponsorship is not available for this position.
Why Join
Early-stage opportunity with significant ownership and impact — you'll shape foundational infrastructure decisions.
Work on genuinely hard distributed systems problems in a production AI context.
Small, experienced team with deep ML and enterprise engineering backgrounds.
Well-funded at the seed stage with strong institutional backing and a clear enterprise customer focus.
Originally posted on Himalayas
Create a free Caio profile to unlock more results and save your role and location preferences.
Unlock free search