Indic speech recognition accuracy: what is actually published

Where the gaps are, and who left them

Four categories of missing number, each with a different consequence for a procurement decision.

How a department should evaluate this for itself

Nothing in the published record substitutes for an evaluation on your own calls. This is the shape one should take.

1. 1

   Sample from your own helpline, not from a benchmark

   Take held-out recordings from the line you intend to automate, across the hours it actually runs, on the telephony path it actually uses. A benchmark file tells you how a model handles that benchmark. Your recordings carry your codec, your line noise, your callers' dialects and your scheme vocabulary, and none of those four are in any published number.

2. 2

   Transcribe to a written protocol, with two passes

   Decide before you start how numerals are written, how English words inside an Indian-language sentence are scripted, how hesitations and false starts are handled, and how unintelligible audio is marked. Have a second transcriber re-do a share of the set blind and measure how far the two disagree. That disagreement is the floor of your evaluation: no model can be measured as more accurate than your reference is consistent. The IndicVoices collection ran a published transcription guideline and a quality-control pass for exactly this reason.

3. 3

   Break the result down by language and by dialect region

   Report every language separately, and within the main language report by district or region where the sample allows. A single pooled number hides the case you are about to fail: the model that scores acceptably overall and badly in the two districts that generate a third of the calls. The published per-language tables show spreads of 20 to 80 points inside a single model, so assume your dialect spread is not zero until you have measured it.

4. 4

   Score numbers and entities as their own metric

   Extract the scheme names, amounts, dates, phone numbers and place names from each reference, and score those separately from the running text. IndicContextEval's named entity error rate is the published precedent for doing this, and the reason to copy it is that entity errors and word errors move independently — a system can improve on one while getting worse on the other.

5. 5

   Report CER alongside WER

   For Indian-language scripts, WER alone overstates the damage from orthographic near-misses and understates nothing. Reporting both tells you whether a model is mis-hearing the word or mis-spelling it, and those have different fixes: the first needs a different model, the second often needs normalisation or a lexicon.

6. 6

   Size the sample and publish the interval

   Enough calls per language that the number stops moving when you add more, and enough per dialect region that the per-region breakdown is not noise. Report the sample size in calls and in minutes of audio alongside every figure, and put a confidence interval on it — a bootstrap over utterances is the standard way and takes minutes. A point estimate with no interval and no sample size is not a result anyone can act on, including you in six months.

7. 7

   Re-run it when anything changes

   A model version, a telephony provider, a new district, a new scheme vocabulary — each one invalidates the measurement. Keep the evaluation set and the scoring script in version control so a re-run costs an afternoon rather than a new project.

Every per-language figure we could find, with what it measures

Sorted by source. Figures in the same column are not comparable across rows unless the dataset, metric and condition all match. A cell reading "no published figure" means we fetched the source and it publishes none.

One caveat that applies to the whole first block: the IndicVoices paper labels its comparison table "Performance" and never states the metric, the normalisation or the versions of USM, Azure, Whisper and MMS it tested. Everyone who cites it, Sarvam included, reads it as WER, and the neighbouring figures are consistent with that. We have reproduced it as published and flagged it rather than quietly calling it WER. Treat the four non-AI4Bharat rows in that block as the weakest-sourced numbers on this page.

The sources, and what class of evidence each one is

Every figure above traces to one of these. They fall into three classes that carry different weight: peer-reviewed papers, vendor-published claims, and independent leaderboards. For Indic ASR the third class is effectively empty, which is why the second class deserves more scrutiny here than it would in English.

Read this first. None of these figures is an AiSewak result. We have published no accuracy number of our own and nothing on this page should be read as one. Each entry below is another organisation's measurement, linked to the page we read it from. This page is a technical reference, not procurement or legal advice — a department should treat it as a reading list, not as a basis for an award.

• 15.0Measured result

  IndicVoices: 7,348 hours across 22 languages and 145 districts, 74% extempore and 17% conversational, with a public train/test split and a per-language comparison of IndicASR against USM, Azure, Whisper and MMS.

  Peer-reviewed. The broadest Indian-language ASR benchmark that exists, and the only one covering all 22 scheduled languages. Its comparison table does not state the metric or the competitor model versions — read the note under the table above before quoting it.

  AI4Bharat, IIT Madras · IndicVoices (arXiv 2403.01926) — Hindi, IndicVoices test set · 2024

• 7.6 → 26.8Measured result

  Vistaar: 59 benchmarks over 12 Indian languages, spanning studio-read, crowd-read, phone-app and telephone-quality spontaneous speech, with the same models scored across all of them.

  Peer-reviewed. The single most useful source on this page for a helpline decision, because it holds the model and the language fixed and varies the recording channel. Gramvaani is described in the paper as telephone-quality speech with a focus on regional and dialectal variation in Hindi.

  AI4Bharat · Vistaar (arXiv 2305.15386) — IndicWhisper, Hindi, IndicTTS versus Gramvaani · 2023

• 8.3 → 9.2Measured result

  IndicSUPERB and the Kathbath corpus: matched clean and noisy test splits across 12 languages, for the same speakers and sentences.

  Peer-reviewed. The cleanest published isolation of added noise as a variable in an Indian-language benchmark, because everything except the noise is held constant.

  AI4Bharat · IndicSUPERB (arXiv 2208.11761) — IndicWav2Vec with LM, Hindi, clean versus noisy unknown-speaker split · 2022

• 10.5Measured result

  IndicWav2Vec: self-supervised pretraining across 40 Indian languages, with published WER on MUCS, MSR and OpenSLR benchmarks and a comparison against the 2021 MUCS leaderboard.

  Peer-reviewed. Useful mainly as the earlier baseline the later Indic models are measured against; its Hindi MUCS figure is read speech, not conversational.

  AI4Bharat · IndicWav2Vec (arXiv 2111.03945) — Hindi, MUCS benchmark, best configuration · 2021

• 21.5Measured result

  Whisper: per-language WER appendix tables for FLEURS, Common Voice 9, MLS and VoxPopuli, for all six model sizes.

  Peer-reviewed, and the most complete per-language disclosure any of the commercial labs has made. It is also the source that shows the failure mode most clearly: the same model scores 21.5 on Hindi and above 100 on Gujarati, Telugu, Malayalam, Punjabi and Sindhi.

  OpenAI · Robust Speech Recognition via Large-Scale Weak Supervision (arXiv 2212.04356) — large-v2, Hindi, FLEURS · 2022

• 10.6Measured result

  MMS: per-language FLEURS WER for 54 languages against Whisper medium and large-v2, plus per-language CER for 102 languages.

  Peer-reviewed, but note that Meta is reporting its own model against a competitor it re-scored itself. It states the normaliser it used, which is what makes the comparison readable at all.

  Meta AI · Scaling Speech Technology to 1,000+ Languages (arXiv 2305.13516) — MMS-1107 LSAH with LM, Hindi, FLEURS test · 2023

• no per-language Indic figureOfficial figure

  Google USM: a 2-billion-parameter multilingual speech model reporting averaged WER and CER across FLEURS-102 and YouTube long-form sets.

  Peer-reviewed, but everything Indic in it is inside a 102-language average. The only per-language Hindi number for USM anywhere in this compilation came from AI4Bharat evaluating it, not from Google.

  Google · Google USM (arXiv 2303.01037) · 2023

• no per-language Indic figureOfficial figure

  SeamlessM4T: a 45% average WER reduction against Whisper across the 77 FLEURS languages both models support, and WER-versus-signal-to-noise curves over four languages.

  Peer-reviewed. Reports ASR quality only as averages over language groups. Its section on input noise is the clearest published statement that error rates rise as signal-to-noise falls, but it is a chart, so there is no figure to cite.

  Meta AI · SeamlessM4T (arXiv 2308.11596) · 2023

• 25.9Measured result

  MUCS 2021: separate multilingual and code-switching ASR tracks, with Hindi-English and Bengali-English test and blind sets, and transliteration-WER reported alongside WER.

  Peer-reviewed. The only published Indian-language benchmark built specifically around code-switching, which is how most urban helpline callers actually speak. Its data table also records the channel compression used per language — 3GP, M4A or PCM — which is rare and worth copying.

  Interspeech 2021 special session · Multilingual and code-switching ASR challenges for low resource Indian languages (arXiv 2104.00235) — best of three organiser baselines, Hindi-English code-switched test set · 2021

• 25.9 NEER vs 16.9 WERMeasured result

  IndicContextEval: 56 hours of natural audio across 23 domains and 8 Indic languages, scoring named entity error rate separately from word error rate at seven levels of supplied context.

  Peer-reviewed. The published precedent for scoring entities as their own metric. For every model tested, entity error runs above overall word error — the gap is 9 points for the best system and 7 for GPT-4o Transcribe.

  AI4Bharat · IndicContextEval (arXiv 2606.19157), Interspeech 2026 — best model, 8 languages pooled · 2026

• 19.31Official figure

  Saaras v3: ~19% WER on the IndicVoices benchmark across 22 languages, and 19.31% on the ten most-spoken subset, compared against GPT-4o Transcribe, Gemini 3 Pro, Deepgram Nova-3 and Scribe v2.

  Vendor-published: Sarvam running a third-party benchmark on its own model against named competitors. The benchmark is independent and the number is specific, which puts it well above a marketing claim; the scoring was still done by the party being scored, and no per-language breakdown is given.

  Sarvam AI · Introducing Saaras V3 — vendor-published, ten-language subset of IndicVoices · 2026

• 16.03Official figure

  Saaras v4: 16.03% WER on IndicContextEval with a native-script entity list supplied, 5.22% language-identification error across 22 languages, and per-dataset Vistaar results across ten languages.

  Vendor-published. The English benchmark table is given as numbers; the Indic per-language results are rendered as an interactive chart with no tabulated values in the page, so they cannot be quoted or checked from the published page.

  Sarvam AI · Introducing Saaras V4 — vendor-published, on the AI4Bharat IndicContextEval benchmark · 2026

• no published figureOfficial figure

  The Bhashini API catalogue lists every ASR service ID, the languages it covers and the institution behind it — IIT Madras, AI4Bharat and others — across the 22 scheduled languages plus regional variants.

  The national platform's own catalogue carries no accuracy figure of any kind. For a department procuring through Bhashini, the numbers behind those service IDs are in the AI4Bharat papers above, not in the catalogue itself.

  Bhashini (MeitY) · Bhashini APIs — available models for usage · fetched Sep 2026

• no published figureOfficial figure

  The IndicConformer 600M model card documents a hybrid CTC and RNN-T conformer across 22 languages, with a required resample to 16 kHz before inference.

  The most widely deployed open Indic ASR model publishes no error rate on its own card. The 16 kHz requirement is the line that matters for telephony: 8 kHz call audio is upsampled to meet it, which adds no information back.

  AI4Bharat · IndicConformer 600M multilingual — Hugging Face model card · fetched Sep 2026

• no published figureOfficial figure

  Deepgram lists Hindi for Nova-3 and Flux and Gujarati and Kannada for Nova-2, and quantifies accuracy only as a relative reduction in English word error rate.

  Vendor documentation. Language support is documented precisely; per-language accuracy is not documented at all.

  Deepgram · Models and languages overview · fetched Sep 2026

• no published figureOfficial figure

  AssemblyAI groups its Universal-2 languages into accuracy tiers explicitly defined by word error rate, and publishes the tier rather than the rate.

  Vendor documentation. A tier is more than nothing — it is an ordering — but it cannot be compared to any figure in the table above.

  AssemblyAI · Supported languages · fetched Sep 2026

• no published figureOfficial figure

  Google's Chirp 3 model page documents hi-IN support and streaming recognition, with no error rate for any language.

  Vendor documentation. The only Google per-language Indic figures in this compilation were produced by AI4Bharat evaluating Google's own services.

  Google Cloud · Chirp 3 model documentation · fetched Sep 2026

• no published figureOfficial figure

  OpenAI's speech-to-text guide lists Hindi, Tamil, Kannada, Marathi, Nepali and Urdu among supported languages for its transcription models, with no accuracy figure.

  Vendor documentation. OpenAI's per-language Hindi numbers exist, but in the 2022 Whisper paper rather than in the docs for the models currently on sale.

  OpenAI · Speech-to-text API guide · fetched Sep 2026

• no published figureOfficial figure

  The Scribe launch post claims the lowest transcription word error rate across FLEURS and Common Voice benchmark tests in 99 languages, naming figures for Italian but none for any Indian language.

  Vendor claim. It names the benchmarks it ran, which is more than most, and then publishes no per-language table for them.

  ElevenLabs · Meet Scribe · fetched Sep 2026

• 0 Indian languagesMeasured result

  The Artificial Analysis speech-to-text word-error-rate index is built from AA-AgentTalk, VoxPopuli-Cleaned and Earnings22-Cleaned, weighted by audio duration across roughly eight hours.

  Independent leaderboard, and the one class of evidence that would settle vendor disagreements. All three of its datasets are English, so for Indic ASR it contributes nothing and no equivalent exists.

  Artificial Analysis · Speech-to-text leaderboard, AA-WER v2 index · fetched Sep 2026

Go deeper

Related pages: AI voice agent for government · Indian-language coverage · Helpline call automation · Grievance redressal · TRAI and ECI rules for outbound calls · Official helpline numbers · AiSewak's AI ethics guidelines