How to Connect a Voice AI Agent to Telegram, WhatsApp, and Your Website

Modern voice models can produce convincing speech with relatively little setup. The engineering work begins when the same agent has to operate across a Telegram chat, a WhatsApp thread, and a microphone widget on a website.

Each channel packages audio differently. Each has its own session rules, API constraints, and acceptable response time. A delay that feels normal after a Telegram voice note can make a live browser conversation feel broken.

This guide explains what a voice AI agent consists of and how spoken input moves from the microphone to the final reply. It also covers the differences between Telegram’s Bot API, WhatsApp’s Cloud API, and a browser-based WebRTC connection. The final sections look at common failure points, practical use cases, and infrastructure requirements once the project moves beyond a local prototype.

What a Voice AI Agent Actually Is

A voice AI agent listens to spoken audio, determines what the person wants, and replies in speech. In a live conversation, this loop needs to happen within one or two seconds. Longer pauses make the interaction feel closer to a walkie-talkie exchange than a natural dialogue.

There are two common architectures.

The first is a cascaded pipeline:

1. Speech-to-text (STT) converts audio into text.
2. A language model decides what to say and which tools to call.
3. Text-to-speech (TTS) turns the answer back into audio.

Each component can be replaced independently. Teams can choose different providers for transcription, reasoning, and speech synthesis. Open frameworks built around this architecture are popular because they provide control over every stage.

The second approach uses a speech-to-speech model. It accepts audio and produces audio directly, without relying on a separate text transcript as the main interface between stages. OpenAI’s Realtime API family, including the GPT-Realtime-2 model released in May 2026, follows this approach.

Speech-to-speech models often preserve tone, pacing, and other vocal details more naturally. In a cascaded system, some of that nuance is lost during transcription and then reconstructed by the TTS engine.

Both architectures also depend on voice activity detection, or VAD. This lightweight process determines when a person starts and stops speaking. Poor VAD settings cause the agent to interrupt users, miss the beginning of a phrase, or wait too long before replying.

These problems become especially visible outside controlled demonstrations. Background conversations, traffic, music, and unstable microphones can quickly expose weaknesses that were not obvious in a quiet office.

How the Pipeline Works, Step by Step

Whichever architecture is behind it, a single turn of conversation moves through roughly the same stages. Here's what happens between someone pressing record and hearing a reply.

Capturing and Detecting Speech

Audio arrives as a raw stream — from a phone's microphone, a Telegram voice note, or a browser's getUserMedia call. VAD watches that stream for the start and end of speech, trimming silence so the rest of the pipeline isn't wasting cycles on dead air. In a live call, this stage also keeps listening while the agent is talking, which is what makes barge-in — a person cutting the agent off mid-sentence — possible at all.

Turning Speech into Meaning

The audio segment gets transcribed, either by a dedicated streaming STT model or as part of a combined speech-to-speech pass. Streaming matters here: waiting for a person to finish an entire sentence before starting transcription adds delay that compounds with everything downstream. Current streaming transcription models return partial results within a few hundred milliseconds, giving the rest of the pipeline a head start.

Deciding What to Say and Do

A language model reasons over the transcript, the conversation history, and whatever tools it has access to — looking up an order, checking a calendar, pulling a customer record. Tool calls are what separate a voice agent that can actually do something from one that only talks. The reply comes back as text, in a cascaded setup, or as a stream of audio tokens directly, in a speech-to-speech setup.

Speaking Back

Text-based replies get synthesized into audio by a TTS engine and streamed out in chunks as they're generated, rather than all at once, so the first sound reaches the listener before the whole sentence has even finished generating. That streaming trick is part of why a well-tuned cascaded pipeline can feel almost as quick as a native speech-to-speech model, even with more steps technically involved.

Handling Interruptions

If VAD detects the person talking again while a reply is still playing, everything downstream — the TTS stream, the LLM generation, sometimes an STT pass still catching up — needs to stop immediately, handing control back to the new input. Frameworks built for this treat every stage as cancellable by design, which is a genuinely different engineering problem than building a chatbot that just waits its turn.

Assembling those stages into a working pipeline is usually a handful of lines once the framework is in place:

def build_voice_pipeline(transport, stt, llm, tts):
    pipeline = Pipeline([
        transport.input(),
        stt,
        llm,
        tts,
        transport.output(),
    ])
    return PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))

Connecting to Telegram

Telegram's Bot API is plain HTTP, and that simplicity is exactly why it's often the easiest channel to wire a voice agent into. Incoming voice notes arrive through a webhook as a message object containing a file_id; a follow-up call to the API resolves that into a downloadable link. From there, the recording goes through STT like any other audio input, and the reply goes back through sendVoice.

The catch is format. Telegram only renders a reply as a proper playable voice bubble — the rounded waveform people expect — if the file is an .ogg container encoded with the OPUS codec (MP3 and M4A also work but lose the native voice-message styling in some clients). Most TTS engines output raw PCM or WAV, so a conversion step sits between the language model's reply and the Telegram API call. ffmpeg handles it in one line:

ffmpeg -i reply.wav -c:a libopus -b:a 32k -vbr on reply.ogg

Voice notes are capped at 50 MB, which in practice is a non-issue for spoken replies — a five-minute Opus recording at that bitrate lands well under 2 MB. Registering the webhook itself is a single call once the server has a public HTTPS endpoint:

curl -X POST "https://api.telegram.org/bot<TOKEN>/setWebhook" -d "url=https://yourserver.example.com/telegram/webhook"

From that point on, every voice note a user sends arrives as a POST request, and every reply goes out the same way a text message would — just with sendVoice instead of sendMessage, and an Opus file instead of a string.

Connecting to WhatsApp

WhatsApp works through Meta's Cloud API, and it treats voice differently from a generic audio attachment. Sending a file with the voice flag set to true makes it render as a native voice note — complete with a play icon, a waveform, and, if the recipient has transcripts turned on, an automatic text version. Leave that flag off and the same file shows up as a plain audio attachment with a download icon instead, which reads noticeably less personal in a support conversation.

The format requirement matches Telegram's core case: an .ogg file with the OPUS codec. Anything else and voice-message transcription on the recipient's side simply fails, even if the audio itself plays fine. There's a size detail worth knowing too — the play icon only appears if the file is 512 KB or smaller; go over that and WhatsApp falls back to a download icon regardless of the voice flag, since it won't auto-fetch a larger file over a metered connection.

Two constraints matter more here than on Telegram. First, a verified WhatsApp Business phone number and an approved Meta Business account are required before any of this works — there's no equivalent of Telegram's instant, anonymous bot creation. Second, the 24-hour customer service window applies: once a person messages your number, you can reply freely with voice, text, or anything else for 24 hours, but starting a new conversation outside that window requires a pre-approved message template, which complicates any workflow built around proactive voice replies. Sending an already-uploaded voice reply looks roughly like this:

{
  "messaging_product": "whatsapp",
  "to": "<customer_phone_number>",
  "type": "audio",
  "audio": {
    "id": "<uploaded_media_id>",
    "voice": true
  }
}

Inbound audio arrives through the same webhook that handles text and image messages, distinguished by its message type, with a media ID your server exchanges for a temporary download URL before running it through STT.

Adding a Voice Agent to Your Website

A website widget is the one channel with no messaging platform standing between the person and the agent — just a browser, a microphone, and whatever backend handles the conversation. That's freeing in some ways and unforgiving in others: expectations for a live, back-and-forth call sit higher than for a voice note that arrives a few seconds late in a chat thread.

Getting audio out of the browser means WebRTC — the getUserMedia API captures the microphone stream, and a client SDK (LiveKit's, Pipecat's, or a direct connection to a speech-to-speech endpoint) streams it to the backend and plays the reply back with minimal buffering. Browsers only grant microphone access on HTTPS or localhost, which is easy to forget until a demo that worked fine on a laptop refuses to even request permission once it's deployed on plain HTTP.

A handful of things need to be in place before a website voice widget works end to end:

• An HTTPS-served page, since browsers block microphone access over plain HTTP
• A backend server reachable in real time — this is typically where a small VPS sits, bridging the browser session and whichever STT, LLM, and TTS providers the agent uses
• A TURN server, or a provider that includes one, so calls still connect from behind restrictive corporate or mobile networks
• A visible fallback, such as a text input or a "call support" link, for visitors who won't grant microphone access at all

Latency expectations here are the tightest of the three channels. A voice note on Telegram arriving two seconds after the transcript would have been ready barely registers; the same two seconds inside a live browser call feels like a bad phone line.

Cloud Servers from €4 / moIntel Xeon Gold 6254 3.1 GHz CPU, SLA 99,9%, 100 Mbps channeltry

Picking an Orchestration Layer

None of the three channels dictate which framework sits in the middle — that choice is really about how much control a team wants versus how fast something needs to be running. A few names come up constantly in 2026 voice-agent projects:

Pipecat and LiveKit Agents both leave the hosting decision to you, which is exactly the point for teams that need to keep recordings and transcripts on infrastructure they control. Vapi and platforms like it trade that control for speed — a reasonable deal for a first prototype, less so once voice data residency or per-minute costs start to matter at scale.

Telegram, WhatsApp, and Website at a Glance

Advantages and Disadvantages

Running one agent across three channels means writing the conversation logic, the tool integrations, and the guardrails exactly once, then adapting only the thin layer that talks to each platform's API. That consistency matters more than it sounds — a customer who gets a different answer on WhatsApp than they got on the website ends up trusting the whole system less, not just the one channel that slipped. It also means usage data from all three surfaces feeds the same analytics and the same improvement loop, so the agent gets better faster than three separate one-off bots ever would.

The cost is that every channel adds its own maintenance surface. Telegram can change bot API behavior with little warning; WhatsApp's business verification and template approval process runs on Meta's timeline, not yours; and a browser widget has to keep working across whatever combination of operating system, browser, and network a visitor happens to have. Testing has to happen on all three, separately, every time the core agent logic changes — a one-line prompt tweak that reads perfectly in a Telegram test can still surface a rare edge case in a live browser call weeks later.

Limitations and Risks

A few risks apply regardless of which framework or channel is in play:

• Latency stacks. Network delay, STT, LLM reasoning, and TTS synthesis each add their own slice of time, and a pipeline that looks fine on paper can still feel sluggish once all four run back to back on a real connection.
• Recorded voice is personal data. Storing raw audio, even briefly, brings the conversation under data protection rules such as GDPR, which means a retention policy and a clear consent notice, not an afterthought.
• WhatsApp's rules bite the unprepared. Messaging outside the 24-hour window without an approved template gets messages rejected outright, which can quietly break an otherwise working voice flow.
• Voice mistakes are harder to walk back than text ones. A misheard word or a hallucinated detail in a spoken reply is gone the instant it's said, with no message left to edit or delete.
• Cost scales with usage in a way that's easy to underestimate. STT, LLM, and TTS are all billed by the minute or by the token, and a genuinely popular agent can run up a bill a text-only chatbot never would.

Where the Complexity Pays Off

Order Status and Support Over Voice Notes

An online store connects the same agent to Telegram and WhatsApp so customers can send a quick voice note — "where's my order from Tuesday" — instead of digging through email confirmations. The agent transcribes the question, calls the store's order API as a tool, and replies with a short voice note giving the delivery estimate. Because both channels feed the same backend, a customer who starts on WhatsApp and later messages on Telegram gets the same tone and the same accurate answer.

A Booking Assistant for a Busy Front Desk

A salon or clinic reception adds a voice widget to its website and a WhatsApp number for the same purpose: taking appointment requests without tying up a phone line. The agent checks a calendar, offers open slots, confirms a booking, and hands off to a human for anything outside routine scheduling — cancellations, complaints, anything that needs judgment a script shouldn't be making.

An Internal Helpdesk That Talks Back

IT teams have started wiring voice agents into internal Telegram bots for password resets, VPN issues, and "is the server down" questions that would otherwise sit in a ticket queue for hours. Employees ask by voice, the agent checks a status dashboard through a tool call, and either resolves the issue on the spot or opens a ticket automatically with the relevant context already attached.

Lead Qualification for Real Estate and Travel

A property or travel agency puts a voice widget on its listings page and a WhatsApp number in its ads. The agent asks a few natural questions — budget, location, timing — and only pulls in a human agent once a lead is qualified, meaning the sales team spends its time on conversations that are actually worth having.

Common Mistakes

• Shipping without testing the audio conversion step under real conditions — a format that plays fine locally can fail silently once it's actually inside a Telegram or WhatsApp voice bubble.
• Forgetting WhatsApp's 24-hour window, then wondering why a proactive voice follow-up never reaches the customer.
• Treating the website widget like a phone line and skipping the fallback for visitors who won't grant microphone access, which is a larger share of traffic than most teams expect.
• Skipping interruption handling, so the agent talks over people who try to correct it mid-sentence — one of the fastest ways to make a voice agent feel unnatural.
• Storing every recording indefinitely with no retention policy, which turns a useful debugging log into a compliance liability.

Where to Run a Voice AI Agent

Whichever channels a voice agent supports, something has to sit between the messaging platforms' webhooks — or the browser's WebRTC connection — and the STT, LLM, and TTS providers doing the actual work: running the pipeline, converting audio formats, holding the webhook endpoints Telegram and WhatsApp both expect to be public and stable. A serverless function can technically do this, but the cold-start delay it introduces lands right in the middle of the latency budget a voice agent can least afford to lose.

A VPS sitting in a data center close to both your users and your AI provider's region tends to be the simpler choice — it stays warm, keeps a stable IP for webhook registration, and gives full control over what's installed, which matters when ffmpeg, a Python runtime, and whichever orchestration framework you picked all need to coexist without competing for resources. Deployment is quick, too: a new server is typically ready within about a minute, and billing that runs in ten-minute increments rather than a flat monthly fee makes it easy to size a server up while testing and back down once traffic settles into a predictable pattern.

For teams handling voice recordings under stricter privacy rules, running the audio pipeline on infrastructure with a known, chosen location — rather than an opaque region picked by a managed platform — also makes the compliance conversation considerably shorter, since a data center in the Netherlands or North America can be selected explicitly instead of assumed.

Conclusion

A voice AI agent that only works in one place is a demo. One that holds a consistent conversation across a Telegram voice note, a WhatsApp thread, and a live browser call is closer to a real product — and getting there mostly means respecting what each channel actually requires: the right audio format, the right session rules, and a backend fast enough that none of it feels like waiting. None of the individual pieces are exotic anymore; the work is in gluing them together carefully and testing each channel on its own terms rather than assuming what worked on one will hold up on the rest.

For more on running the infrastructure behind projects like this, the Serverspace blog covers server setup, networking, and deployment topics in more depth.

Frequently Asked Questions (FAQ)