An AI voice agent that answers a phone call is five parts wired in a loop: a phone leg that gives you audio, speech-to-text (STT), an LLM, text-to-speech (TTS), and a turn-taking layer that decides who is allowed to talk. The hard part is not any single piece. It is making them cooperate in under about a second and a half while a caller on a patchy mobile line talks over your bot.
This post is the architecture map I'd want before starting: what each part does, which choices matter in India, a decision table for build versus platform, and a checklist before the first real call. Numbers come from vendor pages I read on 2026-10-09, and I say where a page did not give me what I needed.
What does a voice agent pipeline look like?
Audio comes in from the phone network, flows through STT into text, the LLM produces a reply, TTS turns it into audio, and the audio goes back down the phone line. Open frameworks model this as a chain of processors passing frames. Pipecat's overview describes exactly that: a transport creates audio frames from the caller's speech, an STT processor emits text frames, a context step combines them with the conversation history, the LLM streams text, TTS converts it to audio, and the transport plays it back. The stages overlap, so the first sentence is spoken while the LLM is still writing the second.
caller phone
| (PSTN / mobile network)
SIP trunk or carrier API ---- 8 kHz telephone audio ----> your server
|
VAD + turn detection (who is talking?)
|
STT (streaming, partial + final)
|
LLM (system prompt + history + tools) <--> CRM, calendar
|
TTS (streaming, sentence by sentence)
|
audio back to the caller, cancellable
Two details that bite later:
- Telephone audio is narrowband. Phone calls are typically 8 kHz audio, so models tuned for 16 kHz or higher can behave worse on real calls than in your laptop demo. Test with recordings from an actual phone call, not a headset.
- Every arrow is a latency cost and a failure point. The companion post on voice agent latency budgets puts numbers on where the time goes.
Which part do you choose first?
Choose the LLM and the telephony last. Start with the language mix, because it limits everything else.
| Decision | What to check | Why it matters |
|---|---|---|
| Languages | Hindi, English, Hinglish, regional languages | STT coverage and TTS voice quality differ per language. Test code-switching with real recordings. |
| Call direction | Inbound (customers call you) or outbound (you call them) | Outbound brings telecom compliance work that inbound mostly avoids. |
| Peak concurrency | Calls at the same moment, not calls per day | Platforms cap concurrent calls by plan. Your own server needs a worker per call. |
| Tool needs | Calendar, CRM, payment link, human transfer | Decides how much your LLM prompt and tool design matter. See how to design tools for AI agents. |
| Data location | Where recordings and transcripts may be stored | Vendors process audio in their own regions. Check each vendor's data-processing terms. |
If you want the prompt side of this first, read voice AI prompting for VAPI, ElevenLabs and Twilio. It covers writing for speech. This post covers the plumbing under it.
STT, LLM and TTS: what to look for in each
STT. You need streaming transcription with partial results and an end-of-utterance signal, not file upload. Deepgram's pricing page lists streaming Nova-3 at $0.0048 per minute (monolingual) and $0.0058 per minute (multilingual), and calls the streaming rates limited-time promotional. ElevenLabs lists Scribe v2 at $0.22 per hour, but that is a batch-style hourly price, so check whether its real-time mode is what you need before you compare. Whichever you pick, send it a recording of your own callers and read the transcript. Names, phone numbers and addresses are where it breaks first.
LLM. A small fast model handles most call flows, and you only pay for a larger one if the conversation needs real reasoning. Anthropic's pricing page on 2026-10-09 lists Claude Haiku 5.5 at $0.10 input and $0.50 output per million tokens (for prompts up to 100,000 tokens) and Sonnet 5.5 at $2 and $10. Time to first token matters more than price here, since the caller is waiting in silence. Measure it from your server region, not from the docs.
TTS. What you care about is time to first audio byte and whether the voice survives Hindi names and numbers. ElevenLabs lists a "v3 Conversational" TTS model at $0.04 per 1,000 characters and Deepgram's Aura-2 at $0.030 per 1,000 characters, both as of the same date. Voice cloning raises consent questions, covered in the ElevenLabs voice cloning ethics and consent checklist.
Turn-taking. This is the part people underestimate. LiveKit's docs describe several modes: a turn-detector model on top of voice activity detection (their default), VAD only for lowest latency, STT-based endpointing, and manual control for push-to-talk. ElevenLabs describes a proprietary turn-taking model inside its agents platform. The fixed-silence approach (wait 500 ms, then reply) is the one that feels robotic and cuts off slow speakers.
How do you connect the phone network?
You need a phone number and a way to get the call audio to your code. Two common routes:
- A carrier or CPaaS API (Twilio is the best-known) that streams call audio over a WebSocket to your server. Twilio's India voice pricing page showed outbound to mobile at $0.0496 per minute and outbound to local numbers at $0.0699 per minute on 2026-10-09. It did not list an India inbound rate on that page; the downloadable price CSVs it links would have it, and I did not open them. Prices are in dollars with no stated currency code.
- A SIP trunk from a telecom provider. Your voice framework or platform registers as the SIP endpoint, and the provider routes calls to it.
For an India-first setup, a provider that handles Indian numbers and the telecom paperwork can save weeks. Vobiz is one such option. Some links on this page are affiliate links; they cost you nothing and I only link tools I'd mention anyway. Its site describes a SIP trunk, bidirectional WebSocket audio in 20 ms frames, real-time call control (hold, transfer, bridge, end), and claims "TRAI + DLT compliant from day one"; you can sign up here. I could not find India per-minute rates on its public pricing page (it showed US rates only), so get a quote for the numbers and routes you actually need. Treat the compliance claim as the vendor's own statement and ask what documents they need from your company.
The honest downsides: a smaller provider has less public track record than a large CPaaS, and you should test call quality at your expected concurrency before committing. Alternatives are Twilio or the telephony integrations built into the voice platforms below.
Build it yourself or use a platform?
| Option | Good for | Watch out for |
|---|---|---|
| Voice platform (ElevenLabs agents, VAPI, similar) | One or two call flows, fast launch, no audio engineering | Per-minute fee on top of everything, less control over vendors and logs. ElevenLabs lists its agents at a $0.08 per minute base rate on 2026-10-09 and the page did not say whether the LLM and telephony are included. |
| Open framework (Pipecat, LiveKit Agents) | Swapping vendors per stage, own logs, cost control at volume | You run workers, scale them and debug audio. Budget real engineering time. |
| Raw WebSocket from carrier to your own code | Learning, or a very narrow need | You rebuild turn detection, interruption and buffering that frameworks already handle. |
| n8n with a telephony node | Simple outbound reminders | Latency and turn-taking are limited. See building calling agents with n8n. |
My rule: start on a platform to find out whether callers will tolerate a bot at all, and move to a framework only when cost or control forces it. If you go the framework route, async Python patterns for LLM apps will save you from blocking the event loop in the middle of a call.
What should the agent do when things go wrong?
Voice failures are social, not technical. Design these before launch:
- Silence for more than a few seconds. Say something ("Still there?") instead of leaving dead air.
- Not understood twice. Offer a human transfer or a callback. Do not loop a third time.
- Tool slow or failing. Say "one moment" before the tool call, and have a spoken fallback if it times out.
- Caller asks for a person. Transfer, every time, without arguing.
- Caller asks "are you a robot?" Answer truthfully.
Scripts and fallback wording are covered in voice agent call scripts for appointment booking; the pieces here are the plumbing that lets those prompts work. For the recording and disclosure questions, start with call recording consent and AI disclosure in India, then confirm with a professional.
Pre-launch checklist
- Record 20 real calls (with the speakers' agreement) and run them through STT. Read every transcript.
- Measure end-to-end response delay on a real phone, not a browser demo.
- Test an interruption mid-sentence 10 times. Does the agent stop talking quickly and keep the context?
- Test a noisy line, a speakerphone and a Hinglish sentence.
- Calculate the cost per minute with your own numbers using the voice agent cost calculator.
- Write the disclosure line the agent says first, and get recording and consent rules checked by a professional who knows Indian telecom and data law.
- Log every call (transcript, tool calls, outcome) so you can review failures. The production checklist for AI agents covers what else to put in place.
What I did not verify
I did not build or place a call for this post. I read the vendor pages linked above in this session, and where a page was incomplete (Vobiz India rates, Twilio India inbound, ElevenLabs latency figures) I said so rather than fill the gap. Vendor prices change, so check the live page before you budget.
