ElevenLabs CTO Pushes Voice Agents In-House to Choke Out Latency

What happened

Piotr Dąbkowski is turning ElevenLabs into its own primary test subject.

The co-founder and chief technology officer of the unicorn voice vendor is deploying conversational AI agents directly into the company's daily development pipeline and customer infrastructure. Rather than treating voice agents as a standalone revenue line for call centres, the engineering group is using them internally to audit systems, handle incoming operational requests, and strip friction out of software shipping cycles.

Speaking to Fortune, Dąbkowski framed the internal rollout as the start of a much wider efficiency sprint. The company wants to compress operational overhead while simultaneously running one of the most compute-heavy voice operations in software. The strategy relies on autonomous voice and text workflows picking up repetitive developer chores, triaging internal tickets, and managing developer documentation lookups before a human engineer touches a keyboard.

That internal automation push coincides with the company's aggressive outward push into the conversational agent market. Over the past six months, ElevenLabs moved from raw file-based text-to-speech generation into full-duplex conversational voice platforms. That pivot demands real-time WebSocket pipelines, sub-400-millisecond response latencies, and continuous model orchestration.

By forcing staff to use the same conversational systems they sell to enterprise developers, Dąbkowski is testing whether his architecture can survive high-stress workflows without blowing out GPU infrastructure budgets.

The background

ElevenLabs built its initial valuation on high-fidelity, expressive audio synthesis. Founded in 2022 by Dąbkowski, a former machine learning engineer at Google, and Mati Staniszewski, an ex-Palantir deployment strategist, the startup captured the consumer and indie developer markets with voice cloning that did not sound like a microwave reading the evening news.

That early lead drew massive capital. The company closed an $80 million Series B round in January 2024 led by Andreessen Horowitz, Nat Friedman, and Daniel Gross, pushing its valuation past the $1.1 billion mark. That cash funded massive GPU procurement across cloud clusters, allowing ElevenLabs to serve millions of creators producing audiobooks, game dialogue, and automated marketing audio.

One-way audio synthesis is an easy engineering problem compared to live telephone conversations. In standard audio production, an application submits a text string to an API, waits two seconds, receives an MP3 buffer, and plays it back. Nobody cares if a podcast narrator takes fifteen hundred milliseconds to render on a server rack in northern Virginia.

Live voice conversation breaks that illusion immediately. The moment you put an AI voice on a telephone line, you encounter three separate bottlenecks:

• Automatic speech recognition must transcribe human speech instantly while ignoring background acoustic noise.
• An orchestration layer or large language model must digest the intent and return structured text within two hundred milliseconds.
• The voice engine must synthesize natural audio chunks using streaming protocols before the human detects an unnatural silence.

If the total round-trip time climbs past seven hundred milliseconds, the interaction falls apart. Callers interrupt, the agent stumbles, and the caller hangs up.

To capture enterprise support contracts, ElevenLabs launched its Conversational AI platform in late 2024. The product bundled speech recognition, language models, and their proprietary low-latency Turbo v2.5 synthesis models into a managed stack. Instead of forcing enterprise engineers to stitch together deepgram sockets, OpenAI inference endpoints, and Twilio media streams with custom Python glue code, ElevenLabs wrapped the entire loop into a single software development kit.

That transition shifted the startup's operational identity. ElevenLabs stopped being a niche media utility for YouTubers and became an enterprise telephony broker competing directly against legacy telecommunications stacks.

What people are saying

The internal agent rollout reflects Dąbkowski's belief that voice platforms cannot scale their headcounts at the same rate they scale server clusters. Running multi-tenant speech models across thousands of simultaneous enterprise phone calls burns cash at a terrifying pace. If a voice startup must hire armies of integration engineers to support custom enterprise accounts, gross margins drop toward traditional IT consultancy levels.

"We see massive potential to automate routine engineering cycles using our own conversational infrastructure, ensuring our developers spend compute on hard architecture rather than manual ticket resolution."

Dąbkowski argues that the efficiency returns from autonomous internal tooling will compound faster than the hardware costs required to run them. The thesis is straightforward: an engineering team that offloads workflow orchestration, integration triage, and continuous testing to automated agents can out-ship legacy communications giants that rely on hundreds of manual software testers.

Not everyone in the speech infrastructure space buys the hype. Rival engineers point out that dogfooding voice agents inside an elite software company is a world away from answering calls for a regional health clinic or a tier-one telecom provider. Enterprise call centres operate on razor-thin margins and show zero tolerance for hallucinated account data, telephony packet drops, or weird acoustic glitches caused by audio compression.

Sceptics also point to the persistent cost of real-time GPU inference. Running high-frequency acoustic models alongside reasoning models on high-bandwidth memory chips like Nvidia's H100s remains an expensive habit. When consumer startups deploy free agents, venture capital covers the electric bill. When enterprise vendors sell agents, the unit economics must pencil out against low-wage human agents in offshore facilities. That leaves little room for inefficient orchestration.

The architectural reality check

Stripping human latency out of software development requires solving the exact same pipeline issues that plague telephone agents. When an engineer asks an internal agent to check a database migration or spin up an isolated staging container, the agent must execute complex tool-calling routines without stalling out.

In standard agent architectures, function calling murders conversation speed. The language model pauses generation, emits a JSON schema payload, waits for an API gateway to authenticate, runs the task against an internal service, receives the raw output, and only then resumes talking. That process can easily chew up three to five seconds of dead air.

ElevenLabs is betting its streaming architecture can mask that latency. By running predictive text-to-speech buffers and speculative tool execution, the company wants agents to speak natural conversational filler while backend workers execute code in the background. If Dąbkowski makes that work reliably for internal developers debugging deployment pipelines, the company can package the same orchestration mechanics for banks, airlines, and logistics firms whose agents must query ancient SQL databases mid-call.

If the system fails internally, it burns developer hours and irritates senior engineers. That instant feedback loop is precisely why Dąbkowski chose to deploy it on his own teams first.

What happens next

The road ahead for ElevenLabs depends on whether internal efficiency gains translate into cheaper, more durable enterprise margins.

Over the next twelve months, the startup faces fierce pressure from hyperscalers. OpenAI is pushing its Realtime API into commercial production, cutting out intermediate text layers through native audio-to-audio multimodal models. Google has integrated low-latency speech pipelines directly into Gemini 2.0 Flash. Cartesia and Deepgram are dropping sub-two-hundred-millisecond synthesis engines that strip out infrastructure costs by running on leaner, specialized neural architectures.

To survive that squeeze, ElevenLabs must prove its managed Conversational AI platform provides better operational stability than raw models rented straight from Microsoft Azure or Amazon Web Services. Dąbkowski's bet is that business customers will pay a premium for a managed orchestration layer that handles interruption handling, noise gating, telephony SIP trunking, and tool execution out of the box.

Watch ElevenLabs' enterprise pricing sheets and API uptime monitors closely over the next two quarters. If internal agent efficiency actually lowers the company's cost to serve, expect aggressive price cuts aimed at choking out smaller open-source speech wrappers. If server costs outpace internal productivity, Dąbkowski will find that running an automated enterprise is much harder than cloning voices in a browser window.