Our Thesis

The world is not made of words.

Language is the most successful technology humans ever invented for compressing the world. We took experience i.e., every smell, every sound, every face, every hesitation before an answer.. and found a way to shrink it into discrete symbols that fit in a paragraph. It works so well that we forget it happened and it was well.. lossy.

We mistake the map for the territory.

Voice AI inherited that mistake. The models that power the Voice AI systems, even today, were built on one implicit assumption: Massive amounts of clean, transcribed, single-language audio data would always be available. And when you have large models at the backbone, you have to follow such laws of scale. For English, this is fair. For three quarters of people alive today, that is unfair - not linguistically, not phonetically!

The consequences have led us substandard speech models. English speakers get production-grade, natural-sounding, emotionally expressive voice AI. Everyone else gets semi-robotic, accent-mangled, code-switch-breaking approximations we cannot deploy without embarrassing themselves in front of our own customers. When the world majorly moves towards more and more B2C voice applications (when consumer AI apps explode in late 2027), this problem will amplify.

For last 4 years, AI's most spectacular results have come from learning to predict the next compressed symbol. The whole internet, tokenized. Out of that simple objective came models that can write essays, debug code, and pass the bar. Speech, Visuals, Videos and most of the rendering for the real-world is where that runs out.

Your voice is warmer when you talk to someone you love. It tightens when you lie. It slows when you're sad. The room you're in is in the recording. So is the time of day, the device, the language you switched to when the English word didn't quite fit. None of this is incidental. All of it has meaning. This is what makes us human.

So we stopped predicting words and started predicting the signal, in a latent space wide enough to hold what a transcript throws away. That got us halfway. A model trained on thin data is uncertain constantly, and what it does with that uncertainty decides whether it sounds human or not. And the output which you see breaking with every other speech model out there - is worst exactly where the data is thinnest, which is every language we actually care about.

We think the foundation needed a breakthrough. Our fix was to change what the network reports. Not a guess, but the evidence behind the guessed utterance outputs. We wrote the whole thing up, derivations and ablations and an honest ledger of what we measured against what we still owe, in our experiements. You can read about it here →

Speech intelligence needs to be built the way intelligence seems to actually work: not by predicting what was said, but by predicting what the world is like - in a latent space rich enough to hold the things words throw away at a fraction of the training cost. What you get from all of this is the new architecture, not a fine-tuned layer that sits on top of it. The evidential encoder basically learns the structure of the speech and its own confidence in that structure at the same time and the latent space it builds is where the meaning sits by continual tokenizer than standard discrete ones which throw the speech metadata away. Our state-of-the-art DCASE 2026 winning domain agnostic leaning algorithms align that latent space across domains and languages - so you bypass the catastrophic forgetting i.e., learning every new language or dialect, you do not need to keep forgetting the older weights. Together they make up layerwise joint embedding predictive architecture.

That's what we're building: A world model for speech. A foundation that learns meaning in a continuous latent space, trained on the most of world's dialects, accents and slangs. Most of the world’s languages have minimal to no speech data to have SOTA models on, thus our foundation model - both compute and data efficient can form a speech AI backbone for the mankind.

02

Research agenda

four directions · one foundation
Direction / A

A world model for speech

Prediction in latent space, not token space. The model learns what a conversation is doing — not just which word comes next. Inspired by the JEPA family of architectures; built for audio from the ground up.

Direction / B

Data Efficiency

The dominant paradigm in any LLM-based model is the sheer scale of the trainable data. For example, speech synthesis requires 100,000+ hours of transcribed audio to reach production quality for a language. Our unified LSM (Latent Speech Model) architecture reaches the same quality threshold with under 5,000 hours. That is a 20× reduction in data requirements. Thus if a certain dialect, having really less available clean & transcribed data, can still its own highly performant synthesis model. Our EDL + DAIL training innovations learn the structure of speech rather than memorizing its surface. These innovations learn the structure of speech rather than memorizing its surface. That's what 10× data efficiency means in practice — and why low-resource languages become viable at all.

Direction / C

Compute Efficiency

Our training regime is TPU optimized. We have also made FA4 training stable on H100s (which was originally written for blackwell). We are moving to smaller, efficient models - eventually getting towards edge deployment. For the voice agent infrastructure, each agent occupies 10X lesser RAM than most other voice platforms and are self-swappable. The design reduces clean data requirements to under 5% of what standard transformer models demand, enables zero-forgetting incremental learning across new languages and dialects, and produces models compact enough for edge deployment without degrading quality.

Direction / D

Duplex at the foundation

Code-mix and noisy environments is not an edge case in most parts of the world — it is the default register of speech. Our corpus and objectives treat Hindi-English mid-sentence switches as first-class signal, not noise to normalize away.

03

The Latent Speech Model

LeJEPA-World core · 4.3 lakh hours · Indic + code-mix
Raw audiowaveform in
LeJEPA-WORLD · the core after Yann LeCun & Fei-Fei Li's world-model thesis
EDL Encoderlearns structure
Latent spacemeaning lives here
DAILaligns decoders
Voice synthesis shipping
Speech recognition next
Speech-to-speech 2027
forward pass gradient ∇ backprop
epoch 47 step 12,480 loss 0.2311

Why a world model, not a token model

LeJEPA-World is the core piece — EDL, the latent space, and DAIL are one self-supervised system in the JEPA lineage. Token prediction optimizes for the transcript; latent prediction optimizes for the signal — prosody, emotion, and timing survive because they were never discarded. The 4 dB MCD number downstream is a direct consequence of this choice.

Why it transfers

Structure learned in one language is largely reusable in the next. Adding a language stops being a full training run and becomes an adaptation — which is the only economics under which 22 scheduled languages ever ship.

04

Results from the MVP

Vaani Lab 🔄 Production Deployments
 

Data Efficiency

We hit production voice quality on ~20x less labelled audio data to match the frontier quality.



Domain Agnostic Learning

Adding new dialects/voices/style doesn't degrade the ones we already shipped.


Compute Efficiency

Our cost per language is nearly flat. Arithmetic Load (T-Flops) Scaling per additional language.


SOTA Phonetic Collision Efficacy

Handling code-switching as efficiently as no-switching without phonetic drift.


05

Model Benchmark study

Apr 2026
Model WER ↓TTFT ↓MCD ↓Training data Architecture
ElevenLabs v3EN-first, fine-tuned 6.1%—†—†≈10× Best in class, better raw English WER. Finetuned for most global languages.
Low performance on code-mix, code-switch and entity/alphanumerical scenarios.
Cartesia Sonic 3EN-first, fine-tuned 7.1%—†—†≈10× State Space Models. Vaani is closer to Sonic-3 on WER despite 10× less data.
Vaani TTSLatent Speech Model 7.2%80 ms4.0 dB1× (ours) Best in class, better raw English WER. Finetuned for most global languages.
Low performance on code-mix and code-switch scenario
Sarvam BulbulEN-first, fine-tuned 7.4%—†—†≈10× Sovereign India only model, not suitable for domain adaptation.
Low performance on code-mix and code-switch scenario
Smallest LightningEN-first, fine-tuned 7.5%—†—†≈10× -

† Methodology. Competitor figures publish with the full study — measured on identical Indic and code-mix test sets, same hardware, same prompts. Methodology, test sets, and replication scripts ship with the paper. Until then we only print our own numbers.


05

Papers & reports

what we publish, we replicate
06

The loop

lab ↔ deployed voice
Lab → Production

Research ships

The same latent foundation we benchmark runs live collections, lead qualification, and service calls for banks and insurers. No lab fork, no demo build — production is the model.

14+enterprise deployments
<400 msp90, deployed pipeline
Production → Lab

Deployment teaches

Every consented production call is a stress test the lab could never write: dialects, interruptions, bad networks, real emotion. Failures route back into training data and evals within the week.

4.3L hrsproprietary Indic corpus
50 TBand compounding
07

The lab

IISc-born · Bengaluru

A 16-person passionate gang. The team started at IISc around 2017 and it waited seven years for the field to catch up. Meet the full team →

08

What's next

the same foundation, more abilities
NowShipping

Voice — Vaani TTS

The first decoder off the LSM. Indic voices, code-mix, native pronunciation of names and numbers, ₹2.5 per 1,000 characters via API.

TRL 7
NextIn research

Listening — recognition & understanding

The same latent space, run in reverse: recognition that hears code-mix, accent, and intent — not just phonemes.

TRL 5
2027Horizon

Speech-to-Speech

No transcript in the middle. Meaning in, meaning out — the conversation as a single continuous signal. This is what the foundation was built for.

TRL 1
09

Work with the lab

two doors in

For researchers & academic labs

Benchmark collaborations, corpus partnerships, and evaluation work on Indic speech. If you study low-resource languages, we probably have data problems you'd find interesting.

nitesh@vaaniresearch.com

For people who want to join

We hire researchers and engineers who want their work in production, not just in proceedings. Founders still do the final interview on every hire.

See open roles

Get the Lab Notes

There are tons of insights we release internally from every brainstorming, model experiment and paper reading session. In case if you wish to be a part, drop your email below - and we'll send you routine lab notes.
No spam, promise!

Prefer to talk? Book a 30-minute call instead.