Evidence, not estimates

Vaani Research Lab. First in a three part series on the evidential architecture behind our Latent Speech Model. Part 2 covers the two stage training framework and the continuous tokenizer. Part 3 presents the experiments.

There is a specific sound that a badly trained speech model makes. Not wrong words. Not glitches. Something worse and harder to name: every syllable is technically correct and the whole thing sounds like it was recorded underwater. Muffled. Airless. The high frequency detail that makes a voice sound like it belongs to a person is just gone.

For years the field treated this as a data problem or a vocoder problem. Train on more hours, use a better neural vocoder, and the mumble recedes. Which is true, but it dodges the actual question: why does the mumble exist at all?

The answer is buried in the loss function, and it is one of the cleaner cases we know of where a single line of math explains an audible artifact. This article works through that math, and then through the fix we use in our foundation model: a network that outputs evidence instead of estimates, and that can tell you, per frame, per channel, how much it actually knows.

1. A regression problem in disguise

Strip away the encoders and attention maps, and a text to speech model is a regression machine. It consumes some conditioning signal and emits a sequence of continuous acoustic frames, usually mel spectrogram vectors $y_t \in \mathbb{R}^d$. Train it with mean squared error and you are minimizing

Here is the line of math that matters. Fix an input $x$ and ask: what prediction $\hat{y}$ minimizes the expected squared error under the true conditional distribution $p(y|x)$? Take the derivative, set it to zero, and you get the textbook answer:

The optimal MSE prediction is the conditional mean. Always. The model is not being lazy or undertrained when it outputs an average. Averaging is the correct solution to the problem MSE poses. If you gave the model infinite data and infinite capacity, it would average better.

For most regression problems this is fine, because the conditional distribution has one mode and the mean sits on it. Speech is not that problem.

Say the target text contains the word "really" with mild emphasis. A human speaker might realize it with a rising pitch contour, or a fall rise, or flat with lengthening. All three are valid. All three appear in any large corpus, produced by different speakers, or by the same speaker on different days. The conditional distribution $p(y|x)$ over acoustic realizations is multimodal, and the modes are far apart in mel space.

Now take the mean of those modes.

The mean of two valid pitch contours is not a third valid pitch contour. It is a compromise that lives in a probability valley, a frame no speaker would ever produce. Do this at every time step, across all $d$ mel channels, and the compromises compound into exactly the artifact you hear: energy smeared across frequencies, sharp harmonic structure blurred out, transients softened. The muffled voice is the sound of the conditional mean.

We call this the oversmoothing trap, and the point worth sitting with is that it cannot be escaped by scale. It is a property of the objective, not of the model. You need a different objective.

2. Why "just predict a Gaussian" doesn't get you out

The obvious next step is to make the model probabilistic. Instead of a point $\hat{y}_t$, output a mean and a variance, $(\mu_t, \sigma_t^2)$, and train with the Gaussian negative log likelihood. Now the network can at least say "I am unsure here" by inflating $\sigma_t$.

This helps, and plenty of systems stop here. Two problems made us keep going.

First, a single Gaussian is still unimodal. It widens around the average instead of committing to a mode. You have upgraded from a confident compromise to a hesitant one.

Second, and this is the one that actually shaped our architecture: a predicted $\sigma_t$ gives you one number for uncertainty, with no way to tell two very different situations apart.

• The frame is genuinely variable. Many valid realizations exist. The data is noisy in an honest way.
• The model is ignorant. It has never heard acoustics like this: an unfamiliar dialect, a corrupted recording, a speaker profile far outside its training distribution.

The first is called aleatoric uncertainty, the second epistemic. A vanilla Gaussian head conflates them into one scalar. For a model trained mostly on clean, curated data, the conflation is a nuisance. For ours, it is fatal, and the reason is specific to how we train.

Our foundation model's first training phase runs on massive amounts of untranscribed speech. Indic languages, code mixed sentences, phone line audio, background noise, overlapping speakers. Nobody has cleaned this corpus, because nobody can afford to clean four hundred thousand hours of it. (Why we train on untranscribed audio at all, and what the two phases look like, is the subject of Part 2.) When a corrupted or wildly out of distribution frame arrives, we need the model to recognize "I don't know what this is" and down weight it, rather than either averaging it into the parameters or dismissing it as ordinary noise. That requires the two kinds of uncertainty to be separated, structurally, in the output.

3. The evidential move: predict the distribution over distributions

Evidential deep learning makes a hierarchy out of the problem. The frame is still Gaussian:

but $\mu_t$ and $\sigma_t^2$ are no longer things the network outputs. They are treated as unknown random variables, and the network instead outputs the parameters of a prior over them. For a Gaussian likelihood with unknown mean and variance, the conjugate choice is the Normal Inverse Gamma distribution:

with joint density

So the network's raw output per frame is a four tuple $\mathbf{e}_t = (\gamma_t, \nu_t, \alpha_t, \beta_t)$, and there is a physical reading for each part:

The "virtual observation" language is not decoration. In Bayesian terms, an NIG prior is exactly what you would arrive at after observing $\nu$ samples informing the mean and $2\alpha$ samples informing the variance. So the network is being asked a strange and useful question. Not "what is the frame?" but "summarize the evidence you have about the frame, as if you had seen it before." A frame it understands well gets a sharp prior backed by a lot of virtual evidence. A frame it has never seen anything like gets a diffuse prior backed by almost none.

The word evidential is earned: evidence is literally the quantity being regressed.

4. What the hierarchy buys you: uncertainty splits in closed form

Here is where the NIG choice stops being a modeling aesthetic and starts paying rent. Because the prior is conjugate, every quantity we care about comes out analytically. No sampling, no variational approximations, no Monte Carlo at training time.

The expected frame:

The aleatoric uncertainty, meaning the model's estimate of the honest variability in the data:

And the epistemic uncertainty, meaning the model's uncertainty about its own estimate of the mean:

Look at the ratio of the two:

The two uncertainties differ by exactly a factor of the evidence. This is the clean structural fact the whole design leans on. A frame can carry large aleatoric variance and still be epistemically certain, if $\nu_t$ is large: "this syllable legitimately has many realizations, and I know that with confidence." Or the reverse: a frame with modest apparent variance but $\nu_t$ near zero is the model admitting it is guessing.

One more closed form matters. Marginalizing out $\mu$ and $\sigma^2$, the predictive distribution over the frame itself is a Student's t:

with $2\alpha_t$ degrees of freedom. When evidence is low, the degrees of freedom are low and the predictive has heavy tails: the model refuses to be surprised by outliers, which is the correct posture toward data it does not understand. As evidence accumulates, the t sharpens toward a Gaussian. The model's willingness to be surprised is itself a learned, per frame quantity.

5. The loss: reward evidence, punish confident mistakes

Training maximizes the marginal likelihood of the ground truth frame under that Student's t predictive. Writing $\Omega = 2\beta(1+\nu)$ and dropping subscripts, the negative log likelihood is

This term alone has a known failure mode: a network can shrink its loss on hard examples by inflating uncertainty everywhere, becoming uniformly and uselessly humble. So the objective adds an evidence regularizer,

giving the full objective

The regularizer reads like a contract. The prediction error $|y^{\text{GT}} - \gamma|$ is multiplied by the total evidence $2\nu + \alpha$. Be wrong with little evidence and the penalty is mild; you claimed ignorance and ignorance was honest. Be wrong while claiming a mountain of evidence and the penalty scales with the claim. Confidence is a bet, and the loss makes the model pay out when it loses. The coefficient $\lambda$ sets the price of overconfidence, trading calibration against sharpness.

This formulation follows the deep evidential regression framework of Amini et al. (2020), which the BELLE line of work first carried into autoregressive TTS. What we do differently is architectural, and it is the subject of Part 2: the evidential head is not a decoration on a supervised TTS model. It is the interface through which an untranscribed audio backbone talks to every downstream decoder we build.

Why this matters for our training data, concretely. Picture a corrupted frame arriving mid stream during pretraining: a phone line artifact, a speaker profile unlike anything in the corpus. A point estimate model has one way to reduce loss on that frame: drag the weights toward it, polluting what it has learned about normal speech. An evidential model has a second option: keep $\gamma$ where the evidence points and lower $\nu$ and $\alpha$ for that frame, honestly recording "I cannot explain this." The gradient pressure on the mean weakens as claimed evidence drops. Our design hypothesis, and the reason we adopted EDL for Phase 1 pretraining, is that this second option acts as a learned, per frame shock absorber against unclean data. Part 3 quantifies how well the shock absorber actually works, including where epistemic spikes land relative to corrupted frames.

6. The evidential head, concretely

Everything above compiles down to a small module sitting on top of the backbone, cheap enough to be boring, which is the point. The backbone hands over a hidden state $\mathbf{h}_t \in \mathbb{R}^D$; a three layer MLP maps it to the evidential four tuple for each of $K$ output channels:

hidden state h_t  (dim D)
      |
[ Linear D -> 512 ]  + GELU + LayerNorm
      |
[ Linear 512 -> 258 ] + GELU + LayerNorm
      |
[ Linear 258 -> 4K ]
      |
      +--> gamma :  identity            (unconstrained)
      +--> nu    :  softplus(.) + 1e-6  (> 0)
      +--> alpha :  softplus(.) + 1.0   (> 1)
      +--> beta  :  softplus(.) + 1e-6  (> 0)

The activation offsets are where several papers' worth of constraints become three lines of code. Softplus keeps $\nu$ and $\beta$ positive. The $+1.0$ on $\alpha$ is load bearing: every closed form in Section 4 divides by $\alpha - 1$, so the aleatoric and epistemic moments only exist when $\alpha > 1$. Bake the constraint into the activation and no training step can ever produce a distribution whose uncertainty is undefined.

The whole head adds under 1.2M parameters to the backbone. The uncertainty machinery is computationally almost free.

7. Sampling: how a frame actually gets made

At inference we could just emit $\gamma_t$ and be done. That would quietly reintroduce a version of the problem we started with: always emitting the expectation flattens the natural variation that makes speech sound alive. Instead, synthesis draws from the hierarchy the model was trained to describe:

import torch

def sample_evidential_frame(gamma, nu, alpha, beta):
    # Stage 1: draw the variance from its Inverse-Gamma prior.
    # (Sampling precision ~ Gamma(shape=alpha, rate=beta) and
    #  inverting is equivalent, and numerically friendlier.)
    precision = torch.distributions.Gamma(alpha, beta).sample()
    variance  = 1.0 / torch.clamp(precision, min=1e-8)

    # Stage 2: draw the frame around gamma, scaled by the evidence.
    std = torch.sqrt(variance / nu)
    return torch.distributions.Normal(gamma, std).sample()

Two things are worth noticing.

The spread of the draw is $\sigma^2/\nu$, variance divided by evidence. Where the model has strong evidence, samples cluster tightly around $\gamma$ and output is stable. Where evidence is thin, the sampler explores. Sampling temperature is not a knob we set globally; the model modulates it per frame, per channel, through its own learned confidence.

And deliberately, the sampler stops one level up the hierarchy. The full generative story would add a third draw, $y_t \sim \mathcal{N}(\mu_t, \sigma_t^2)$, injecting the aleatoric noise on top. We sample the mean level and stop, because adding the full data noise term to every frame would lay a noise floor across the spectrogram, and the vocoder would faithfully render it as hiss. The structured variation lives at the $\mu$ level; the aleatoric term is where recording noise lives, and we would rather model it than reproduce it.

8. What we are not claiming

A caveat before Part 2, because careful readers will raise it and they should. There is an active line of work questioning whether evidential methods yield faithful epistemic uncertainty in the strict Bayesian sense; Meinert et al. (2023) and Bengs et al. (2022) are good entry points. We are not staking the architecture on a philosophical claim about true posteriors. The claims we make are operational: the evidence outputs separate honest data variability from model ignorance well enough to protect pretraining on unclean audio, to flag corrupted frames, and to drive per frame sampling. Those are measurable claims, and Part 3 measures them.

What the hierarchy has already given us in this article is a model whose output answers three questions instead of one. What do you expect? How variable is the truth? And how much do you actually know? For a foundation model whose entire premise is learning from oceans of audio nobody transcribed or cleaned, that third answer is the one that makes the other two trustworthy.

Part 2 puts this head inside the machine it was built for: a two stage architecture that pretrains on untranscribed audio, a continuous tokenizer that sidesteps discrete codebooks, and a training setup with no forced aligner anywhere in it.

Vaani AI Research, Bengaluru. The Latent Speech Model and the experiments referenced here are covered across this series; methodology and replication details ship with the accompanying papers.

References

1. Amini, A., Schwarting, W., Soleimany, A., Rus, D. Deep Evidential Regression. NeurIPS 2020.
2. Sensoy, M., Kaplan, L., Kandemir, M. Evidential Deep Learning to Quantify Classification Uncertainty. NeurIPS 2018.
3. Meinert, N., Gawlikowski, J., Lavin, A. The Unreasonable Effectiveness of Deep Evidential Regression. AAAI 2023.
4. Bengs, V., Hüllermeier, E., Waegeman, W. Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation. NeurIPS 2022.
5. BELLE: Bayesian Evidential Learning for continuous autoregressive TTS. (Full citation per the version referenced in our internal notes.)

Vaani Research Lab. Second in a three part series on the evidential architecture behind our Latent Speech Model. Part 1 built the evidential head: a network that outputs Normal Inverse Gamma evidence instead of point estimates. Part 3 presents the experiments in full.

Part 1 ended with a small module that regresses evidence. On its own, that head is a nice trick you could bolt onto any TTS model. This part is about the machine it was actually built for, and the design pressure that shaped every piece of it: almost none of our audio has transcripts.

That single fact, obvious to anyone who has tried to build speech systems for Indian languages, quietly breaks most of the standard TTS recipe. It breaks the data assumption, because supervised pipelines want paired text and audio, and for most Indic languages that resource barely exists. It breaks the tokenizer, because discrete codebooks were designed for a compression problem we do not have. And it breaks the forced aligner, an external tool sitting at the base of pipelines like FastSpeech 2, which cannot run at all when there is no transcript to align against.

So this article is really three connected stories: how we split training into two stages so untranscribed audio does most of the work, why we made our tokenizer continuous when most of the field went discrete, and how the evidential machinery from Part 1 lets us delete the forced aligner instead of replacing it.

1. The data asymmetry

Two numbers frame everything. Transcribed, studio quality, paired text audio data for a major Indic language: typically hundreds of hours, sometimes less, expensive to grow. Raw untranscribed speech in the same language: effectively unbounded. Call recordings, broadcast archives, open corpora, field collections. Noisy, multi speaker, code mixed, and nobody wrote down what was said.

A model trained end to end on paired data inherits the small number. Every hour of learning about speech itself, its rhythms, its phonetics, how a voice carries a sentence, has to come out of the same scarce budget that also teaches the model to read.

The fix is old wisdom applied to a new place: decouple what can be learned without labels from what cannot. A child spends years absorbing the sound structure of a language before anyone shows them an alphabet. Reading gets attached to an existing speech competence, and attaching it is cheap by comparison. Our training mirrors that split exactly.

Phase 1, audio to audio. A backbone learns to represent and reconstruct speech from raw untranscribed audio. No text anywhere in the loop. This is where the volume lives, and where the evidential head earns its keep, since this data is exactly the unclean, out of distribution mess that Part 1's uncertainty machinery was designed to absorb.

Phase 2, text to audio. A text encoder learns to drive the pretrained backbone, using the small paired dataset. The backbone is frozen or updated at a deliberately tiny learning rate ($10^{-5}$), so the scarce paired data spends its budget on alignment, not on relearning what speech sounds like.

The claim we care about, quantified in Part 3, is that this split lets the full system reach competitive naturalness using roughly a tenth of the paired data that end to end baselines need. The rest of this article is about the three components that make the split actually work.

2. The tokenizer question, and why we went continuous

Before a backbone can model speech, something has to turn waveforms into a representation worth modeling. The field's dominant answer since SoundStream and EnCodec is discrete: quantize the audio into tokens from a finite codebook, then model token sequences the way language models model text. It is a seductive answer, because it lets speech borrow the entire transformer LM toolchain.

We think it is the wrong answer for synthesis, and the reason is visible in a single picture.

A pitch contour is a continuous trajectory. A codebook is a finite set of points. Mapping the first onto the second produces a staircase, and the gap between curve and staircase is a residual that no amount of training removes, because it is not a property of the network. It is a property of finiteness.

We can state this carefully. It is worth being precise about what is proved, what is argued, and what is measured, so here is the honest version.

The reconstruction error of a discrete system carries an additive term controlled by the quantization residual $\|E(\mathbf{x}) - \mathbf{z}_q\|$, which is bounded away from zero whenever the encoder's output distribution is continuous and $K$ is finite. A continuous latent $\mathbf{z} = E(\mathbf{x})$ has no such term; its reconstruction error is limited by network capacity alone.

Two honesty notes, because a careful reader will raise both.

First, this is a statement about an error term, not a theorem that discrete systems always sound worse. A large enough codebook pushes the residual below perceptual relevance for many signals; that is why EnCodec sounds good as a codec. The argument is about where each system's ceiling comes from. Discrete systems have a ceiling set by codebook geometry. Continuous systems have a ceiling set by model capacity, which scales with compute. We would rather be limited by the thing that improves every year.

Second, discreteness is not free to discard. A codebook buys you a bitrate, compatibility with token LMs, and a compression story. Those are real benefits for transmission and for audio language modeling. Our claim is narrower: for a synthesis intermediate representation, where nothing needs to be transmitted or compressed, the quantization residual buys nothing and costs fidelity precisely on the continuous micro structure, F0 trajectories, micro pitch, breathiness, that makes speech sound alive. Those cues are the first casualties of the staircase, and they are the ones our whole architecture exists to preserve.

Where the fully formal version stops: proving a tight lower bound on perceptual error for a specific trained system would require assumptions about the encoder's output distribution that we cannot verify. So we treat the proposition as a design argument, and we look to evidence for its consequence. Today that evidence is external: published discrete codebook systems trail continuous ones on naturalness at comparable or greater data scale, consistent with the proposition's prediction. The cleanest test, our own architecture ablated discrete against itself with all else frozen, is designed and ships with the LSM paper; Part 3 lays out that evidential ledger honestly.

Our tokenizer is therefore a continuous VAE in the XCodec lineage: encoder, continuous latent with a Gaussian posterior $q_\phi(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\mu_\phi(\mathbf{x}), \sigma_\phi(\mathbf{x}))$, decoder, trained on the usual ELBO,

with adversarial and feature matching losses added for reconstruction sharpness.

2.1 Teaching the tokenizer to mean something

A tokenizer trained only to reconstruct learns acoustics without semantics. It can rebuild the waveform, but its latent space does not organize around what was said, which is what the downstream model needs. Discrete approaches hit the same wall from the other side: HuBERT style semantic tokens capture phonetics and throw away the voice.

We get both by construction, in a three stage schedule.

S0, acoustic reconstruction. Encoder and decoder train on reconstruction with GAN and KL losses:

S1, semantic distillation. With the encoder frozen, a semantic module learns to predict the features of a frozen pretrained ASR encoder (Whisper) from the acoustic latent:

This is the quiet trick of the whole pipeline. Whisper's encoder has already absorbed hundreds of thousands of hours of multilingual supervision. Distilling its geometry into our latent imports that phonetic knowledge without us ever needing the transcripts it was trained on. The teacher's supervision was paid for once, by someone else, and transfers through features alone.

S2, unified training. Encoder frozen, semantic module and decoder train jointly so the unified feature both reconstructs audio and preserves the distilled semantic structure:

The output is a single continuous representation that carries the voice and the message at once, which is exactly the substrate the backbone needs.

3. Phase 1: learning speech without reading

With the tokenizer fixed, Phase 1 trains the backbone: a transformer over continuous latents, closed by the evidential head from Part 1, trained on raw untranscribed audio at scale.

Three supervision signals shape the backbone at once.

Reconstruction through evidence. The backbone predicts NIG parameters for the target latents, trained with the evidential loss from Part 1. This is where the uncertainty machinery meets the dirty data it was built for. On a corrupted frame, the cheapest way to reduce loss is to lower claimed evidence rather than corrupt the mean, so unclean audio gets absorbed instead of amplified.

Multi teacher semantic anchoring. Frozen ASR teachers (Whisper, HuBERT) provide feature targets that keep the semantic side of the latent honest. Different teachers have different failure modes; distilling from several averages out individual biases.

Disentanglement pressure. The latent is split into a semantic component and an acoustic component, with the training setup arranged so the semantic space carries content and sheds speaker identity. The design target is mutual information near zero between the two, $\text{MI}(Z_{\text{sem}}, Z_{\text{acou}}) \approx 0$. Whether that target is achieved is an empirical question, and it has one of the cleanest answers in our whole experimental suite: a probe trained to recover speaker identity from the semantic latent performs at 1%, against a 16.67% random chance baseline, while a text identity probe reads content at 100%. Part 3 walks through the probe methodology; here it is enough to say the separation is not aspirational.

Why disentanglement matters this much: Phase 1 data contains thousands of speakers we will never want to clone, in recording conditions we never want to reproduce. If speaker identity leaks into the semantic space, every downstream generation drags along a smear of accidental voices. A clean split means the backbone learned speech, not speakers, and voice becomes something you add back deliberately.

4. Phase 2: attaching text, distribution to distribution

Phase 2 is where most systems would retreat to ordinary supervised training. We almost do. A text encoder consumes tokens, and its output must drive the frozen backbone. The interesting decision is what "must match" means.

The lazy answer is point matching: make the text encoder's output vectors close to the audio backbone's vectors in L2. But Part 1's entire argument was that point estimates average away multimodality, and the same failure reappears here one level up. A sentence does not correspond to one point in latent space. It corresponds to a distribution over valid deliveries.

So Phase 2 matches distributions. The frozen audio backbone, run on the ground truth audio, produces a target NIG distribution $T = \text{NIG}(\gamma_T, \nu_T, \alpha_T, \beta_T)$ per frame, per channel. The text encoder predicts its own NIG distribution $P$. The loss is the KL divergence between them, which for the NIG family exists in closed form. Writing it per channel:

where $\psi$ is the digamma function. The decomposition is exact: the first bracket is the expected KL between the conditional Gaussians over the mean, and the remaining terms are the KL between the Inverse Gamma beliefs over the variance. (Readers rederiving this against earlier deep evidential literature should note the final term, $\alpha_P(\beta_T - \beta_P)/\beta_P$; it vanishes when the scale beliefs agree, which makes it easy to drop by accident, but its gradient with respect to $\beta_P$ matters away from convergence.)

Read the formula as a teaching contract. The text encoder is not told "output this vector." It is told: place your mean where the audio evidence placed it, weighted by how confident the audio evidence was, and adopt the same beliefs about variance. Where the audio target was itself uncertain, aleatorically variable delivery, say, the text encoder is permitted to be uncertain there too, at low cost. Uncertainty flows through the interface instead of being crushed into a point.

5. The forced aligner is not replaced. It is unnecessary.

There is one more standard component conspicuously absent from everything above, and its absence is a consequence, not a decision we had to engineer separately.

Supervised TTS pipelines in the FastSpeech 2 lineage need frame level durations: which audio frames belong to which phoneme. Those come from an external forced aligner, usually the Montreal Forced Aligner, and the dependency is a quiet catastrophe for our setting, three times over. MFA needs transcripts, and Phase 1 has none. MFA needs grapheme to phoneme lexicons, which barely exist for many Indic dialects, and degrade badly on code mixed speech where a sentence slides between Hindi and English mid clause. And MFA assumes clean audio; background noise and overlapping speakers make it skip words or collapse outright.

Notice that nothing in our pipeline ever asked for durations. Phase 1 is audio to audio; its "alignment" is the identity. Phase 2 matches distributions over latent frames, and the correspondence between text positions and frames is carried by the model's own attention over the evidential field, soft, probabilistic, learned, rather than imported as hard labels from an external tool. Where a forced aligner asserts "phoneme $k$ owns frames 412 through 447," the evidential field expresses a graded density: how strongly each frame's latent distribution is shaped by each piece of content. Uncertain regions, boundaries, noise, ambiguous coarticulation, show up as low evidence rather than as brittle hard edges placed in the wrong spot.

The natural worry is whether soft alignment converges, whether, without hard duration supervision, the learned field actually tightens onto the true correspondence rather than drifting. We do not have a formal proof, and we will not pretend otherwise. What we have is a working bound that organizes our intuition, stated here with its honest label:

so that as accumulated evidence grows, $\nu_t, \alpha_t \to \infty$, the evidential field converges to the true alignment density without frame level supervision.

The intuition behind each factor: $\mathcal{H}(Y|X)$ is the genuine conditional ambiguity of the mapping, the part of alignment that is uncertain even in principle; $1/\nu_t$ says accumulated mean evidence shrinks the model's excess uncertainty proportionally, mirroring how the epistemic variance $\beta/(\nu(\alpha-1))$ decays in $\nu$; the $\mathcal{O}(1/(\alpha_t-1))$ remainder tracks the settling of the variance belief. Making this rigorous, with the right regularity conditions on the encoder and data distribution, is open, and we would genuinely welcome a proof or a counterexample. What we can say today is empirical: our models train to convergence with no aligner in the loop, on data where MFA cannot run at all, and the resulting alignments survive the quantitative audits in Part 3.

For orientation, the landscape of alignment mechanisms:

6. Streaming, since production is the point

Everything above would be academic if inference were slow, and speech has the least forgiving latency budget in AI: a voice agent that pauses for a second is broken in a way users can feel.

The architecture streams natively because nothing in it requires global context. Text arrives in chunks and slides through the text encoder in small windows. The backbone emits evidential parameters $(\gamma_t, \nu_t, \alpha_t, \beta_t)$ frame by frame. The two stage sampler from Part 1 draws each latent frame in closed form, two cheap distribution draws, no iterative refinement, no diffusion loop. A lightweight neural vocoder (HiFi-GAN or BigVGAN class) converts frames to 24 kHz PCM as they arrive. First audio leaves the system while the sentence is still being generated.

It is worth pausing on why the evidential choice helps here rather than costing us. Probabilistic synthesis usually taxes latency, since sampling from a learned distribution tends to mean iteration. The NIG family's conjugacy is what makes the tax vanish: the predictive is analytic, so sampling is two draws from textbook distributions with parameters the network already emitted. The uncertainty machinery that protects training is, at inference, nearly free.

7. What we are claiming, and what comes next

Compressing this part into its three commitments. Untranscribed audio can carry most of the training burden if the interface between phases is distributional rather than point valued. Continuous latents avoid a quantization floor that discrete codebooks cannot, an argument we state as a proposition and verify empirically rather than overclaim as a theorem. And alignment supervision is not a requirement of TTS but an artifact of architectures that had no way to represent uncertainty about correspondence; give the model that vocabulary and the aligner has nothing left to do.

Each claim has a number attached to it, and Part 3 is where the numbers live: the convergence behavior of the distributional KL objective across training runs, the disentanglement probe audit in full, the oversmoothing and naturalness benchmarks against discrete and point estimate baselines, and an honest accounting of where the system does not yet win.

Vaani AI Research, Bengaluru. Architecture diagrams for both phases, the tokenizer training schedule, and the S³-AT extension proposal are documented in the accompanying technical materials.

References

1. Amini, A., Schwarting, W., Soleimany, A., Rus, D. Deep Evidential Regression. NeurIPS 2020.
2. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., Tagliasacchi, M. SoundStream: An End-to-End Neural Audio Codec. IEEE/ACM TASLP 2021.
3. Défossez, A., Copet, J., Synnaeve, G., Adi, Y. High Fidelity Neural Audio Compression. TMLR 2023. (EnCodec)
4. Hsu, W.-N., et al. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM TASLP 2021.
5. Radford, A., et al. Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023. (Whisper)
6. Ren, Y., et al. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. ICLR 2021.
7. McAuliffe, M., et al. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. Interspeech 2017.
8. Ye, Z., et al. Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. AAAI 2025. (X-Codec)
9. Kong, J., Kim, J., Bae, J. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS 2020.
10. Lee, S., et al. BigVGAN: A Universal Neural Vocoder with Large-Scale Training. ICLR 2023.

Vaani Research Lab. Final part of a three part series on the evidential architecture behind our Latent Speech Model. Part 1 built the evidential head. Part 2 built the machine around it: two stage training, continuous tokens, no forced aligner.

Research blogs have a credibility problem, and it is mostly self inflicted. The genre rewards a certain kind of number: large, round, favorable, floating free of the experiment that produced it. Readers who work in the field have learned to discount accordingly.

So before any results, the ground rules for this part. Every number below comes with its provenance: what was measured, at what scale, by whom. Some of the evidence is ours, from experiments we designed to de-risk this architecture before scaling it. Some of it is inherited, published results from the research lineage we build on, and we will say so plainly, because the difference between "we measured" and "the literature reports" is exactly the difference readers use to decide whether a lab can be trusted. And some things are not yet evidenced at all; those get named too, as debts.

What the experiments below were designed to answer is narrow and deliberate. Before committing serious compute to a foundation model, we needed to know whether the two riskiest assumptions in Parts 1 and 2 survive contact with data. Can a small evidential head actually learn the distributional structure of a latent audio space, rather than collapsing to point estimates with decorative error bars? And does the encoder actually separate what is said from who said it, which is the property the entire untranscribed data strategy stands on? Architecture validation, not product benchmarking. The distinction matters for how far the numbers should be generalized, and we will hold ourselves to it.

1. Experiment 1: can the head learn a distribution at all?

The cheapest way to waste a pretraining budget is to discover, three weeks in, that your output layer cannot represent what your data contains. Part 1 claimed the NIG head learns evidence, not estimates. Experiment 1 tests that claim in isolation, on a controlled task where failure would be unambiguous.

Setup. A standalone sampling module, the three layer architecture from Part 1, is trained to predict the four NIG parameters $(\gamma, \nu, \alpha, \beta)$ per latent dimension over a synthetic latent audio space: 16 outputs covering a four dimensional latent. The dataset is 1,000,000 samples, split 700,000 / 200,000 / 100,000 for train, validation, and test. The objective combines the evidential NLL from Part 1 with MSE and MAE terms between targets and sampled outputs, and training runs for 10 epochs.

One design detail deserves attention, because it is the part that makes this a test of distribution learning rather than curve fitting: each predicted NIG distribution is asked to explain two independent target realizations, $y_1$ and $y_2$, drawn from the same underlying conditional. A point estimator faces an impossible task here; no single vector is close to two different draws at once, and the best it can do is average them, which is precisely the oversmoothing failure from Part 1. The only way to score well on both realizations simultaneously is to place a well shaped distribution over them. The task is rigged so that distributional competence is the only winning strategy.

Results.

The headline number is a best validation loss of 0.0023 on the combined objective. We resist the urge to present that number as self interpreting; the absolute value of a combined loss depends on target scale, and a small number alone proves little. Three things in the curves matter more than the number.

First, convergence is real and stable. The stronger run descends smoothly across the full 1,500 steps with no divergence and, more importantly for an evidential model, no collapse: the model does not escape the NLL by inflating uncertainty everywhere, which is the known failure mode the evidence regularizer exists to prevent, and which we specifically watched for.

Second, the periodic spikes visible in both curves are epoch boundaries, the optimizer meeting reshuffled data, and the model recovers from each within a handful of steps. Recovery speed after distribution shift is a small preview of the property we care about at scale, absorbing surprises without destabilizing.

Third, and most honestly: the two runs differ by roughly 5× in final loss, from configuration alone, same data, same architecture. We are showing the weaker run on purpose. Evidential objectives have real hyperparameter sensitivity, the $\lambda$ trade off from Part 1 has teeth, and a reader planning to build on this should know that the first configuration they try may be the red curve, not the black one.

What this validates, and what it does not. It validates that the evidential head, as specified, learns the distributional structure of a latent space with high precision and without collapse, at a scale of a million samples. It does not validate end to end speech quality; nothing in this experiment produced audio. It is a load test of the foundation, not a photograph of the building.

2. Experiment 2: does the encoder know the difference between *what* and *who*?

Part 2 staked the entire untranscribed data strategy on disentanglement: the semantic latent must carry content and shed speaker identity, or every hour of unclean multi speaker audio pollutes the model with accidental voices. Design pressure toward that separation is built into training, but designed for is not the same as achieved. Experiment 2 audits it.

Setup. The audit uses linear probing's stronger cousin: MLP classifier probes trained on frozen audio encoder embeddings, with stratified k-fold cross validation. Two probes, one question each.

• Probe A (semantic): predict the text identity of the utterance from the embedding. If content survives in the latent, this probe should succeed.
• Probe B (acoustic): predict the speaker identity, six speakers, so random chance is 16.67%. If speaker information has been stripped, this probe should fail, and fail to chance.

The hypothesis structure is worth making explicit because it is what makes this an audit rather than a demo. The null hypothesis $H_0$ is that the encoder is not disentangled, that a capable classifier can recover speaker identity from the semantic embedding at above chance rates. We are trying to reject our own architecture's central promise, with a probe given every advantage: nonlinear capacity, cross validation, direct access to the embeddings.

Results.

The semantic probe reads content out of the embedding essentially perfectly. The speaker probe does not merely fail to beat chance; it lands below it, at 1%. We flag that detail rather than glossing it, because below chance accuracy is a finding with its own texture: it means the probe found no stable speaker signal to exploit, and what little structure it latched onto during training failed to generalize across folds, the signature of an embedding that is not merely weakly informative about speakers but systematically uninformative. $H_0$ is rejected. The encoder passed the audit we designed to break it.

Caveats, stated not buried. Six speakers is a small identity space; a floor result at six speakers is necessary, not sufficient, for a floor result at six thousand, and the scaled audit ships with the foundation model work. And probe accuracy is an operational proxy for the mutual information target $\text{MI}(Z_{\text{sem}}, Z_{\text{acou}}) \approx 0$ from Part 2, not a measurement of it; probes lower bound extractable information, they do not upper bound what exists. What the experiment establishes, precisely, is that speaker identity is not recoverable from the semantic latent by a well equipped adversary at this scale. For the practical question at stake, whether untranscribed multi speaker audio is safe fuel, that is the operative property.

3. The inherited evidence, labeled as such

Now the part of the results section most blogs would blur, and we will not.

Parts 1 and 2 referenced a naturalness benchmark: MOS around 4.2 at a fraction of competitors' training data. Those numbers are not our measurements. They are the published results of BELLE, the Bayesian evidential TTS work whose mathematical engine, NIG evidential sampling with multi teacher distillation, our architecture builds on directly. Presenting the lineage's numbers as our own would be the exact credibility failure this series opened by criticizing, so here they are, correctly attributed, alongside the published baselines they were evaluated against:

Read as a frontier, the table says something specific. Every step up that ladder is a step toward modeling the distribution of speech rather than a point estimate of it: MSE averaging at the bottom, discrete tokens that trade fidelity for tractability in the middle, deterministic flows above that, and calibrated evidence at the top, reaching statistical parity with ground truth (4.21 against 4.20) on the least data of any system in the comparison. That published frontier, combined with our two owned experiments confirming that the evidential machinery works as specified on our stack, is the evidence base for the bet. What it is not, yet, is a measurement of our foundation model.

Which brings us to the discrete versus continuous question one final time. Part 2 argued the quantization floor as an empirically supported proposition. The empirical support today is exactly what the table shows: published discrete codebook systems trailing continuous ones on naturalness at comparable or greater data scale, consistent with the proposition's prediction. What we do not yet have is the cleanest possible version of that evidence, a controlled ablation, our architecture with the tokenizer swapped discrete against itself, all else frozen. That ablation is designed and belongs to the LSM paper, and until it runs, the proposition keeps the epistemic status we assigned it: an argument with corroborating external evidence, not a closed case.

4. The ledger

The honest summary of three articles, sorted by epistemic status.

Measured, ours. The evidential head learns latent distributional structure to 0.0023 validation loss over a million samples, without uncertainty collapse, with quantified sensitivity to configuration. The encoder strips speaker identity to floor level (1% against 16.67% chance) while preserving content perfectly, at six speaker scale.

Published, inherited. Evidential TTS reaches ground truth level MOS at 5,000 hours in the BELLE work our architecture extends. Discrete and point estimate baselines trail on naturalness at greater data scale.

Argued, corroborated, not proved. The quantization floor proposition (Part 2), consistent with the published frontier, awaiting our controlled ablation. The alignment convergence conjecture (Part 2), consistent with the empirical fact that our models train to convergence with no aligner on data where MFA cannot run, awaiting formal treatment or a counterexample, and we remain sincerely in the market for either.

Owed. Foundation scale MOS, WER, and speaker similarity for the Latent Speech Model itself, measured under a protocol we will publish before the numbers, so the evaluation design cannot quietly bend around the results. The scaled disentanglement audit. The discrete ablation. These ship with the LSM paper and the Vaani Benchmark methodology, both targeted for Q4 2026, and this series is, among other things, a public commitment to that.

5. Where this goes

The experiments above validated an architecture at proof of concept scale. The work now is scale, and one research direction deserves a closing note because it follows from everything in this series.

The pipeline in Part 2 still contains one structural asymmetry: the tokenizer is trained first and frozen, which means the representation cannot co-adapt with the model that consumes it. Our S³-AT proposal, the streaming semantic acoustic tokenizer, dissolves that boundary: a single streaming tokenizer trained jointly with the backbone, semantic and acoustic factorization built in from the first gradient step, designed for the duplex, real time setting our production systems live in. It is a proposal, not a result, and it is labeled accordingly, but it is where the evidential thread pulls next.

Three articles ago this series opened with a claim about a sound: that the mumble of a badly trained speech model is the audible signature of a loss function averaging away everything alive in a voice. The argument since has been one long chain from that observation. Replace estimates with evidence (Part 1). Build the training machine so evidence is the interface, and the aligner and the codebook fall away as unnecessary (Part 2). Then audit the chain link by link, own what you measured, attribute what you inherited, and publish your debts (Part 3).

A model that knows what it doesn't know turned out to be the easy part. The discipline is a lab that does.

Vaani AI Research, Bengaluru. The LSM paper and the Vaani Benchmark methodology are targeted for Q4 2026. Replication details for the experiments in this part ship with the accompanying technical materials.

References

1. Amini, A., Schwarting, W., Soleimany, A., Rus, D. Deep Evidential Regression. NeurIPS 2020.
2. BELLE: Bayesian Evidential Learning for continuous autoregressive TTS. (Full citation per the version referenced in our internal notes.)
3. Shen, J., et al. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. ICASSP 2018. (Tacotron 2)
4. Défossez, A., Copet, J., Synnaeve, G., Adi, Y. High Fidelity Neural Audio Compression. TMLR 2023. (EnCodec)
5. Belinkov, Y. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 2022.
6. Meinert, N., Gawlikowski, J., Lavin, A. The Unreasonable Effectiveness of Deep Evidential Regression. AAAI 2023.