Real Time Voice AI Engineer

Non-Negotiable Requirements

Real-time AI Systems:

• Production experience with real-time voice AI / conversational agents: turn-taking, barge-in, interruption handling, sub-second latency
• Streaming media / WebRTC, or frameworks built on it (LiveKit, Daily, Pipecat, or similar)
• Orchestrating multiple AI vendors into one coherent, session-level experience, with fallbacks and graceful degradation
• Demonstrated latency optimization — you have profiled a real-time pipeline and driven a number down
• Cost engineering — you think in cost-per-user-per-minute, not just "does it work"
• Production LLM integration, including the deterministic scaffolding around the model

Engineering Foundation:

• Strong backend / serverless architecture
• Firebase Firestore — you have designed production data models, not just queried them
• Production mobile delivery; React Native strongly preferred (shipped to both App Store and Google Play)

How You Work:

• Live, direct communication. Daily overlap with US Eastern hours.
• You state hard technical truths plainly. We would far rather hear "that's not possible yet — here's what is" than be told what we want to hear.
• You are resourceful about building what doesn't exist yet.

Projects

Sugar Holdings is a founder-led company building a stealth consumer AI product. Small, scrappy, moving fast, with big dreams.

We believe the next generation of consumer AI won’t be chatbots — it will be presences. Something persistent and personal that remembers you, grows with you, and shows up when you need it. Most AI products reset every time you open them. We’re building the opposite: a relationship that compounds.

The honest challenge: the technology we depend on is expensive, immature, and fast-moving. Some of what we need doesn’t exist off the shelf — vendors are building it with us. Making it feel seamless, affordable, and reliable is the hardest problem in the company, and it’s why this role exists.

What You'll Own

• Real-Time Orchestration Layer: A backend router and state machine that determines, in real time, which technology is active, and switches between them invisibly. This is the architectural core.
• Session Lifecycle Management: Starting, stopping, and resuming live streaming sessions on demand, under strict latency and cost constraints.
• Real-time Voice Pipeline: Wake-phrase, hands-free, sub-1.5-second end-to-end response in noisy ambient conditions, with real turn-taking and interruption handling.
• LLM Harness: Middleware enforcing persona consistency, structured memory injection, safety guardrails, and server-side deterministic rules. Not prompt engineering; a production system.
• Persistent Memory Architecture: Structured session records and living summaries that compound across sessions and days, injected into every subsequent interaction.
• Camera Perception Layer: Including confidence gating and graceful fallback. Reliability over false precision.
• Full Mobile Product: React Native (iOS + Android), Firebase, serverless backend, subscriptions, push, analytics, error monitoring, GDPR, and store submission.
• Vendor Co-Development: Working directly with our vendors' engineering teams on capabilities that don't yet exist off the shelf.

Engagement

• Target Launch: October 2026
• Type: Full-time or contract. Fixed-price milestones or hourly for the right candidate. Potential to grow into a long-term technical leadership role.
• Location: Remote, with meaningful US Eastern overlap.

Strong Advantages

• Real-time avatar / talking-head platform experience
• Wake-word detection / always-on listening in noisy environments
• Vision/perception work where knowing the model's limits mattered
• LLM harness or middleware architecture
• RevenueCat