Voice AI Mastery Series - Part 1: Getting Interruption Management Right

I'm a Software Engineer working on Voice AI & Agents at Numeo.ai. This is Part 1 of a 3-part series where I'll share what I've learned building voice AI systems that actually work in production.

After months of building voice AI agents at Numeo.ai, I've learned that the difference between a frustrating AI call and a smooth conversation often comes down to one thing: timing. When should your AI speak? When should it wait? Get this wrong, and users will hang up before they hear how smart your AI actually is.

Why Timing Matters More Than You Think

Picture this: you're talking to a voice AI, you pause to think for a second, and BAM – the AI jumps in with a response. You try to speak again, it keeps talking, and suddenly you're both talking over each other. Sound familiar? This is what happens when interruption management goes wrong.

I've seen this kill user adoption faster than any other technical issue. Users will forgive a slightly robotic voice, but they won't forgive an AI that constantly interrupts them or feels sluggish to respond.

The Technical Foundation

Before diving into the timing stuff, let's talk about Voice Activity Detection (VAD). We use Silero VAD in our LiveKit setup, and it's pretty solid. VAD tells you when someone is speaking versus when they're quiet, but that's just the beginning.

Here's what we configure:

Min Speech Duration: 100ms (how long someone needs to speak before we consider it real speech)
Min Silence Duration: 200-300ms (how long silence lasts before we think they're done)
Activation Threshold: 0.7 (confidence level to trigger speech detection)

These are important, but the real magic happens in turn detection.

The Two Critical Parameters

Turn detection is where your AI decides "okay, now it's my turn to speak." There are two parameters that will make or break your conversation flow:

Min Endpoint: How Patient Your AI Is

This is the minimum time your AI waits before thinking the user is done talking. Too short, and your AI interrupts natural pauses. Too long, and conversations feel painfully slow.

I've found these work well:

800ms for most conversations (customer service, general chat)
600ms for fast-paced stuff (logistics, quick info exchange)
1000ms+ for complex topics (healthcare, where people need time to think)

Max Endpoint: Your Safety Net

This is the maximum time your AI will wait before it definitely starts talking, no matter what. This prevents your AI from waiting forever when someone trails off or gets distracted.

Here's the thing: when you hear both the human and AI talking at the same time, it's usually because the max endpoint timing is off.

The Geography Problem

This is where things get tricky. Network latency varies wildly depending on where your users are calling from, and your interruption settings need to account for this.

The math problem: If your min endpoint is 200ms, but your network latency is 300ms, your AI will never wait for the user to finish speaking. By the time the silence is detected and processed, the user has probably started talking again.

Real numbers I've seen:

East Coast to West Coast: 60-80ms
International calls: 150-300ms
Bad connections: 400ms+

What we're working on:

Measuring round-trip time when calls connect to get baseline latency
Adjusting parameters based on location - East Coast vs West Coast calls need different settings
Using multiple SIP providers and routing based on call destination for better performance
Building a tracing system with OpenTelemetry and SigNoz dashboards to monitor conversation quality metrics in real-time

Measuring What Matters

Our team lead Beka always tells us: "when you don't measure, you can't optimize." So we've built a monitoring system using OpenTelemetry and SigNoz to track conversation quality metrics in real-time.

Key metrics we track:

LLM TTFT (Time to First Token): How quickly our language model starts responding (we're seeing ~773ms)
TTS TTFB (Time to First Byte): How fast our text-to-speech generates audio (~369ms)
Network RTT: Round-trip time for different network paths (220-222ms range)
Conversation duration: Total length of each interaction

Having these dashboards has been game-changing. You can see immediately when parameters need adjustment for different regions or user types.

Platform-Specific Stuff

LiveKit Setup

Here's what our LiveKit turn detection looks like:

turn_detection = EnglishModel(
    min_endpoint_silence=0.8,  # 800ms - works well for most cases
    max_endpoint_silence=2.0,  # 2000ms - prevents infinite waiting
    prediction_threshold=0.7,  # confidence threshold
)

SIP Provider Considerations

If you're working with RingCentral, Twilio, or similar:

SIP adds 50-100ms latency
WebRTC conversion adds more processing time
Plan for 100-200ms additional latency from media processing

Common Problems and Fixes

The "Chatty AI" Problem

Symptom: Your AI jumps in after every tiny pause Fix: Increase min_endpoint, maybe add some confidence-based logic

The "Sluggish AI" Problem

Symptom: Long awkward pauses that make conversations feel unnatural Fix: Reduce min_endpoint, but be careful not to go too low

The "Ping-Pong" Problem

Symptom: AI and human keep interrupting each other Fix: Focus on max_endpoint optimization - this is usually the culprit

What's Next

Getting interruption management right is foundational. Once you nail this, you can focus on making your AI sound more natural (Part 2) and actually have good conversations (Part 3).

In my experience, spending time on this upfront saves you from user complaints later. A well-timed AI with a mediocre voice beats a perfectly synthesized voice that interrupts constantly.

The key is testing with real users, not just your development team. We all get used to our AI's quirks, but first-time users will notice timing issues immediately.

Next up: Part 2 - Making Your AI Sound Actually Human (without spending a fortune on voice cloning)