Software Development Engineer III

Valiant Harbor International is seeking a Software Development Engineer III (SDE III) to support the Director’s Office at the Advanced Research Projects Agency for Health (ARPA-H). This candidate will contribute to the General Research Assistant and Content Engine (GRACE) development team in building the next generation of agentic AI to transform how ARPA-H Program Managers accelerate research, make decisions, and ship products at scale. GRACE is ARPA-H’s production AI assistant, and ARPA-H’s intention is to evolve it into an ecosystem of autonomous, multi-agent systems. This is a full-time, remote position. The candidate must be able to travel within the U.S.

Key Responsibilities

• Design and build agentic AI systems and orchestration:
• Design and build GRACE's core agentic workflows (e.g., multi-step reasoning, planning, memory, and tool-use across single and multi-agent systems).
• Implement and evolve A2A communication patterns at the application layer, enabling GRACE agents to collaborate and hand off tasks.
• Build and maintain the tool-calling layer (tool definitions, input/output schemas, error handling, retry logic, and result formatting).
• Manage the MCP client-side integration.
• Design multi-agent workflows that are reliable, observable, and debuggable in production.
• Facilitate LLM application development:
• Own LLM orchestration at the application layer (prompt construction, context management, model selection logic, and response parsing).
• Build and maintain RAG features (query formulation, result ranking, citation grounding, and hallucination mitigation).
• Implement and iterate on prompt engineering patterns and system prompts across OpenAI GPT, Anthropic Claude, and Google Gemini.
• Manage context window budgets (truncate, summarize, paginate, etc.) and build the logic that makes those decisions correctly.
• Build evaluation pipelines for LLM quality (grounding assessment, regression testing, safety checks, and A/B experimentation on prompt and model changes).
• Manage prompts and pipelines that are cost-efficient without sacrificing output quality.
• Manage features and products:
• Translate ambiguous product requirements into clear technical designs for fast shipment.
• Build new GRACE capabilities end-to-end (from backend application logic through to the API contract the frontend).
• Rapidly prototype new agentic features, run experiments, collect data, and iterate based on real user behavior.
• Perform oversight and quality assessments; write tests, handle edge cases, and make sure your features degrade gracefully when upstream dependencies fail.
• Manage reliability and collaboration with internal/external partners:
• Instrument agentic workflows with tracing, logging, and metrics so failures are diagnosable and regressions are caught before users report them.
• Define and monitor application-level SLOs: tool call success rates, response quality, and latency from the user's perspective.
• Build fallback and guardrail logic for AI services.
• Write production-quality code: readable, tested, reviewed, and documented.
• Work closely with the infra engineer to understand system-level constraints and design application behavior that respects them.
• Participate actively in design reviews, mentor other engineers, communicate technical decision clearly to both engineers and non-engineers.
  

Required Qualifications

• Bachelor's or Master's in Computer Science, Software Engineering, or related field, or equivalent practical experience.
• 7+ years of professional software engineering experience building and operating production systems.
• Proven experience in high-velocity environments shipping complex products end-to-end.
• Strong proficiency in Python and at least one other backend language; familiarity with modern backend frameworks and async patterns.
• Solid understanding of algorithms, data structures, APIs, and software design patterns.
• Experience building and operating systems on major cloud platforms (AWS, GCP, or Azure).
• Experience with containerization and working within CI/CD pipelines.
• Self-starter with a high bar and high sense of urgency; you do not wait to be told what to do next.

Preferred Qualifications

• Hands-on experience building production systems on top of LLMs: tool-calling, RAG, multi-step reasoning, and context management.
• Experience with multi-agent (A2A) architectures and orchestration frameworks in production.
• Familiarity with MCP at the client/consumer layer: how agents discover and invoke tools via MCP.
• Strong intuition for prompt engineering and LLM behavior across model families.
• Experience building LLM evaluation and regression testing pipelines.
• Cost-per-query awareness, context budget management, and prompt efficiency.
• Track record in startup or early-stage environments: 0-to-1 product building, comfort with ambiguity, high sense of urgency.
• Experience in big tech building customer-facing AI platforms or developer tools at scale.
• Background in security-conscious engineering (input validation, output sanitization, audit logging, and responsible AI guardrails).

Salary Range: Negotiable

EEO Statement: Valiant Harbor International, LLC is an Equal Opportunity/Affirmative Action employer. Valiant Harbor International prohibits discrimination with respect to the hiring or promotion of individuals, conditions of employment, disciplinary and discharge practices, or any other aspect of employment on the basis of sex, race, color, age, national origin, religion, disability, marital status, sexual orientation, gender identity, pregnancy, veteran status, or any other protected class. If you are an individual with a disability and require a reasonable accommodation to complete any part of the application process, or are limited in the ability or unable to access or use this online application process and need an alternative method for applying, you may contact (202) 417-6705 for assistance.

This is a full time position