Acceptance-Testing a Helpline Language by Language

Executive Summary

A tender that asks for "22 languages" buys a list. A helpline launches one language at a time, and each launch is a decision someone must be able to defend: why this language, on these call types, from this date. The published research shows why one pooled accuracy figure cannot support that decision. Inside a single model, the spread across Indian languages runs from about 13 to over 40 on the same benchmark.

This guide is for the state IT department, NIC unit or helpline cell that runs an IVR and a team of agents today. Callers press a digit for a language and an agent types the complaint into the portal. The change proposed here is a written acceptance record for each language, with a fixed sample, a pass rule agreed in advance, a named scorer and a launch tier. A language that fails stays on the existing menu and agents.

Executive Callout Three parts of the record decide most outcomes: how many calls were tested, what counts as a critical failure, and who scored them. Our two earlier guides cover which engine to choose and how to test code-switched speech. This one covers the step after both: turning test results into a launch decision. AiSewak is an implementer. It is not the Bhashini portal, claims no endorsement or partnership from it, and publishes no accuracy figure of its own.

Introduction: A Launch Is a Decision, Not a List

Picture the review meeting before go-live. The vendor shows Hindi working well and says Tamil and Marathi are supported. An officer asks whether Tamil is ready for the 8 pm surge, from a mobile in a district office. Nobody has a number that answers it, because the demo ran on a quiet line in one language.

The fix is procedural. Each language is tested separately, on the department's own recordings, against a pass rule written before the first call is scored. The multilingual voice agent page states the principle that a language ships when it passes its own set. This guide sets out what that set and that pass rule contain.

Why One Pooled Number Cannot Launch a Language

AI4Bharat's IndicVoices benchmark covers 7,348 hours of speech in 22 languages and 145 districts. Its comparison table scores several systems on each language. The paper labels the table "Performance" and does not name the metric; the values behave like word error rate, so we read them that way (lower is better). We read the paper on 19 October 2026.

Language	AI4Bharat IndicASR	Google USM
Punjabi	12.9	35.7
Hindi	15.0	20.5
Marathi	18.2	30.7
Telugu	26.8	40.3
Tamil	31.2	58.9
Malayalam	40.5	78.2

One model, one benchmark, and a spread of roughly three to one from the best language to the worst. Google's system scores 99.8 on Sindhi in the same table. The authors built IndicASR and trained it on this data, so its lead is expected. The table is a research benchmark, not a helpline measurement, and none of these values should appear in a tender as a requirement.

The channel adds a second spread. In the Vistaar benchmark, one Hindi model scored 7.6 on studio-read speech and 26.8 on telephone-quality speech, and Google's system scored 18.3 and 59.9. That comparison also mixes dialect with channel, so read it as a warning about direction. Our engine comparison guide covers both papers.

A pooled average describes none of your callers. Either the language passes on your line or it does not.

The Acceptance Record: Five Parts

Keep one record per language, in a form a non-technical officer can read.

Part	What it states	Who owns it
1. Test set	Held-out recordings from the department's own line, by district, call type and device	Helpline cell
2. Scoring	Transcript rules, critical-failure list, entity and number scores	Independent scorer
3. Sample and interval	Number of calls, number of distinct callers, interval on every figure	Evaluator
4. Pass rule	Thresholds written before scoring, per call type	Department head
5. Launch tier and re-test triggers	The tier granted, the date, what forces a re-test	Department head

Part 1: The test set

Use recordings from the line the agent will answer, across the hours it runs, on the carrier paths callers use. Split them by district, call type and device, and hold them out from any tuning. The code-switching guide lists five groups of calls that belong in the set.

Check retention first. Call audio carries names, addresses and complaint details. The DPDP Act's consent and penalty sections commence on 13 May 2027, and the Act is only partly in force today. Our guide to grievance data retention sets out what may be kept and for how long.

Part 2: Scoring

Word error rate alone cannot launch a helpline language. A caller who gets the wrong status read out does not care that the transcript scored well. Agree a list of critical failures per call type before scoring starts. For a grievance line it usually contains:

a complaint or reference number filed or read back incorrectly;
a complaint routed to the wrong department;
a status given for the wrong case;
a call that ends without a human offer after the caller asked for one.

Score numbers and named entities as their own measure, and record every critical failure with the recording ID. The scorer should be a speaker of that language and register who does not work for the vendor. A vendor scoring its own system is the one arrangement that cannot be audited.

Part 3: How many calls

Sample size is where most acceptance tests quietly fail. If a test finds zero critical failures in n independent calls, the 95% upper bound on the true failure rate is about 3 divided by n. This is the "rule of three" described by Hanley and Lippman-Hand in a 1983 JAMA paper. The table below is that arithmetic, not a measurement.

Clean calls, zero critical failures	Failure rate could still be as high as
50	6%
100	3%
150	2%
300	1%
600	0.5%

A department that wants to be 95% sure a language has a critical-failure rate under 2% needs about 150 clean calls in that language. A failure found during the test raises the number needed, so plan for the sample to grow. Two cautions follow.

Calls are not fully independent. Fifty calls from five callers are not fifty independent trials. Count distinct callers and districts, and recruit widely.
Every subgroup needs its own sample. If the department wants a claim for one district, 150 calls pooled across the state do not provide it. Either test that district or do not make the claim.

For word error rate, report an interval as well. Bisani and Ney's bootstrap method, published at ICASSP in 2004, resamples the test utterances and is cheap to run. A figure without a sample size and an interval is not a result.

Part 4: The pass rule

Write thresholds before scoring, per call type, and file them in the record. Set them from the baseline measured on the department's current line; nothing published offers a defensible target for your callers. Use separate rules for routine calls (status, reference number) and sensitive ones (anything that closes or escalates a case).

Part 5: Launch tier and re-test triggers

The tier is the output, covered next.

The Launch Ladder

A language need not be all-or-nothing. A tier ladder lets a language contribute while it earns trust.

Tier	What the voice layer does	Entry condition	Moves up when
0. Hold	Nothing. Callers stay on the existing menu and agents	Fails the pass rule	Fix and re-test on a fresh sample
1. Assisted	Transcribes and drafts the complaint; an agent reviews every filing	Meets the pass rule on routine calls	Agent corrections stay below the department's threshold
2. Guarded	Completes routine call types alone; any failed read-back or uncertain turn goes to a person in the same language where possible	Meets the pass rule on routine calls with a clean critical-failure list	Handoff share and read-back failures stay inside thresholds
3. Full	Handles defined call types end to end, with a person always reachable	Sustained results at tier 2, and a pass on sensitive call types	Reviewed by the department head at each re-test

The thresholds in the right-hand column belong to the department. Nobody can set them honestly before the baseline exists.

After Launch: Monitoring and Re-Test Triggers

Acceptance describes one moment. Re-test the affected language when an engine or model version changes, when the telecom provider or codec path changes, when a new district or scheme vocabulary arrives, or when a new call type joins the agent's scope.

Between re-tests, a weekly report by language shows the completion rate, the share of calls handed to a person and the read-back failure rate. A language that breaches its threshold for two reporting periods in a row drops one tier automatically. Writing that rule before launch removes the argument later.

What Changes in the Helpline's Workflow

Today the caller presses a digit for a language, and the call reaches an agent who may or may not speak it well. Coverage is the number of languages on the menu, and performance is one blended figure.

With a per-language record, the menu offers the voice layer only in languages that hold a tier and routes the rest to agents as before. The helpline cell reports tier by language to the department head. The portal remains the system of record, and named officers keep every decision on a case.

Roadmap

Weeks 1 to 2. Pick one district and no more than three languages. Collect recordings with the proper authority, confirm retention rules and recruit scorers from outside the vendor.
Weeks 3 to 4. Transcribe, write the critical-failure list and agree each pass rule.
Weeks 5 to 6. Run the test, compute intervals and grant tiers. A language that fails goes to tier 0 with a written reason.
After go-live. Run the weekly report, apply the demotion rule and re-test on any trigger.

Risks and Mitigation

Testing on the vendor's recordings. The result describes the demo. Use the department's own line.
A sample too small to mean anything. Fifty clean calls leave room for a 6% failure rate.
Self-scoring. Appoint an independent speaker of the register.
Thresholds set after seeing results. Write the pass rule first, date it and file it.
Silent model changes. A re-test trigger tied to version changes catches what a calendar misses.

Key Takeaways

One model's error rate varies about threefold across Indian languages on the same benchmark, so a pooled figure cannot launch any single language.
Keep a five-part acceptance record per language: test set, scoring, sample and interval, pass rule, tier.
With zero critical failures in n independent calls, the 95% upper bound on the failure rate is about 3/n. About 150 clean calls support a claim of under 2%.
Use a tier ladder, so a language can start in an assisted role and earn autonomy.
Write the demotion rule and the re-test triggers before launch.

Conclusion

Neither a vendor demo nor a research table can tell a helpline cell how an engine will handle its callers. The cell can decide, in writing and before testing, what passing means in each language, test its own calls with a sample large enough for the claim, and publish the tier each language has earned.

Government leaders exploring AI-powered citizen engagement can begin with a focused pilot in one department or constituency to validate impact before scaling statewide. Aisewak helps public institutions deploy multilingual Voice AI solutions designed specifically for Indian governance. The multilingual voice agent page describes the language coverage and acceptance testing approach in more detail.

FAQ

What is a per-language acceptance test for a helpline?

It is a written test, run on recordings from the department's own line, that decides whether a voice agent may handle calls in one language. It has a fixed sample, a pass rule agreed before scoring and a named scorer. Each language gets its own record and its own result.

How many calls do we need per language?

It depends on the claim. With zero critical failures in n independent calls, the 95% upper bound on the failure rate is about 3/n, so 100 clean calls support only "under about 3%". About 150 support "under 2%", and 300 support "under 1%". Any failure found raises the number needed.

Can one language's pass count for another?

No. In the IndicVoices benchmark, one model's error rate runs from 12.9 for Punjabi to 40.5 for Malayalam. Test every language separately, and every subgroup you intend to make a claim about.

Who should score the calls?

A speaker of that language and register who does not work for the vendor. The scorer's marks, with recording IDs for every critical failure, go into the record. A vendor scoring its own system cannot be audited.

What counts as a critical failure?

An error that changes what happens to the caller's case, such as a wrong reference number, a wrong department or a wrong status read out. The department lists these per call type before scoring, since a transcript can score well and still carry one.

What happens when a language fails?

It stays at tier 0. Callers keep the existing menu and agents, and the record states the reason. The language is re-tested on a fresh sample after fixes, never on the same recordings.

How often should a language be re-tested?

Whenever something that affects it changes: an engine or model version, the telecom path, a new district or vocabulary, or a new call type. Between re-tests, a weekly report by language shows handoff and read-back failure rates.

Do test recordings fall under the DPDP Act?

Call audio carries personal data, so check retention and consent before collecting it. The Act's consent and penalty sections commence on 13 May 2027, and the Act is only partly in force today. See the retention guide for what a department may keep.

Schema Markup Suggestions

Article: headline, description, datePublished and dateModified of 2026-10-19, publisher AiSewak.
FAQPage: the eight Q&A pairs above.
HowTo (optional): the five-part acceptance record and the four-step roadmap.

Suggested External References

AI4Bharat, IndicVoices — arXiv 2403.01926, 2024, Table 7, read 19 October 2026
AI4Bharat, Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR — arXiv 2305.15386, 2023, Table 3
Bisani and Ney, Bootstrap estimates for confidence intervals in ASR performance evaluation — ICASSP 2004, doi 10.1109/ICASSP.2004.1326009
Hanley and Lippman-Hand, If nothing goes wrong, is everything alright? — JAMA, 1983 (the rule of three)
Digital Personal Data Protection Act, 2023 — commencement of consent and penalty provisions on 13 May 2027

"22 languages supported" is a list, not a launch decision. A guide for state helpline cells: a five-part acceptance record per language, how many calls the claim needs, and a tier ladder from hold to full.

LinkedIn Executive Summary

A state helpline cell choosing to add Tamil, Marathi or Bengali to a voice agent needs a decision it can defend, not a vendor's language list.

The published research shows why one accuracy figure cannot make that decision. On AI4Bharat's IndicVoices benchmark, a single model's error rate runs from 12.9 for Punjabi to 40.5 for Malayalam. Telephone-quality speech adds a second penalty on top.

The practical alternative is a written acceptance record per language: held-out recordings from the department's own line, a list of critical failures agreed before scoring, an independent scorer, and a sample sized for the claim. With no failures in n independent calls, the failure rate could still be as high as 3/n at 95% confidence, so 150 clean calls support a claim of under 2%.

Results become a tier: hold, assisted, guarded or full. Re-test when an engine, a telecom path, a district or a call type changes.

AI Search Optimization Summary

Entities: Bhashini, Digital India Bhashini Division, MeitY, AI4Bharat, IndicVoices, Vistaar, IndicASR, Google USM, ICASSP, JAMA, DPDP Act, NIC, Aisewak

Topics: per-language acceptance testing for helplines, launch criteria for multilingual voice agents, sample size for acceptance tests, rule of three, bootstrap confidence intervals for word error rate, critical failure definitions, tiered language launch

Semantic keywords: helpline go-live criteria, language-by-language launch, held-out call recordings, independent scorer, pass rule written before testing, Hold Assisted Guarded Full tiers, re-test triggers, demotion rule, read-back failure rate