Sarvam AI

I’m a product manager at CallSense, a speech analytics startup in Pune. I’ve been there since January 2023. I own the QA-scoring product. It scores recorded support calls in Hindi, English and Hinglish against each customer’s checklist.

That product depends on speech-to-text. So I designed and ran a test of three vendors on 2,000 labelled calls from four customers. The company switched vendors as a result.

I also defined our ‘usable transcript’ measure: calls where QA auditors didn’t need to listen to the audio. It’s now in every customer’s weekly report.

Before that I spent two and a half years at Shopwise in Bengaluru. I joined as a data analyst on returns. In 2021 I moved into product, on search relevance. I left to join a startup and own a product end to end. That was CallSense.

I started at PayLane in Mumbai, doing analytics for a consumer lending product. I was there nine months. The company shut down its lending product, and my role went with it.

So the thread is data, then measuring quality, then speech. At CallSense I’ve learnt what I can from the buying side. I’ve judged speech models on real calls. Now I want to work on the models themselves: deciding what gets fixed, and when a model is ready.

Three reasons. First, the side of the table. Since January 2023 I’ve bought speech models and reported what they get wrong. I want to be on the side that decides what gets fixed, and when a model is ready.

Second, you build for Indian languages first. On our calls, people switch between Hindi and English mid-sentence. A model built for that from the start is the one I want to work on.

Third, the audio. You work with banks and insurers, and your voice agents run on calls. That’s the audio I know: support calls in Hindi, English and Hinglish, for customers that include a bank. I’ve spent my time judging speech models on that kind of call.

Saaras and Bulbul are where those decisions get made. That’s why I want to work on them myself.

At CallSense our QA-scoring product depends on a speech-to-text vendor. I designed and ran a test of three vendors to see which worked best on our real calls.

I used 2,000 labelled calls from four customers. Taking calls from four customers meant no single customer’s audio decided the result. The data team built the labelling tool. I designed the test and ran it.

Every vendor ran on the same calls, so we compared them like for like. The company switched vendors as a result.

For Saaras across Indian languages, I’d change four things.

One, slice the set. I’d sample by language, region and accent, noise level and channel. Every slice needs enough calls to report on its own.

Two, report per slice. An average across languages can hide one language that’s getting worse.

Three, check the words that matter: names, numbers, amounts and dates. Those decide whether a bank can use the transcript.

Four, add CER, character error rate, next to WER, word error rate, for Indic scripts. And agree text normalisation first, so Hinglish spelling variants don’t count as errors.

I’d also keep a held-out set that never goes into training, refreshed on a fixed schedule.

Usable transcript asks one thing: did a QA auditor need to listen to the audio? WER counts every wrong word the same. So the two track each other only loosely.

A Hinglish transcript can score badly on WER because of spelling. ‘Acha’ and ‘accha’. Or Hindi written in Roman script in one place and Devanagari in another. The auditor doesn’t mind, so the transcript is still usable.

The reverse happens too. WER can look fine while one wrong digit in an account number makes the call unusable.

Marathi has a second problem. Words are long and carry suffixes. One wrong suffix counts the whole word as wrong. CER gives a fairer picture there.

So I’d report to each side differently. Research gets WER and CER by language, accent and noise, after agreed normalisation. Plus error rates on names and numbers.

A customer gets a measure tied to their task: usable transcripts, or account numbers transcribed correctly.

And I’d keep the two linked. If WER improves and the customer measure doesn’t move, the test set is missing what customers hit.

My hands-on work is on speech recognition. I haven’t run a TTS evaluation. What carries over is designing a human-judged quality measure and building labelled test sets from real calls. Here’s how I’d set it up for code-mixed Bulbul.

One, the sentences. I’d build the set from what voice agents say: Hindi-English mixed sentences, names, loan amounts, dates and account numbers. Every version gets scored on the same set.

Two, the raters. Native Hindi speakers who use Hinglish every day, from more than one region. I’d mix in a few samples of known quality to catch careless raters.

Three, two kinds of test. MOS, the mean opinion score listeners give for naturalness, tells me where a voice stands. Side-by-side preference tells me whether the new version beats the old one. For a release decision, side-by-side is the sharper test.

Four, sample size. I’d start with a few hundred sentences and several raters on each. Then I’d check the confidence interval. If it’s wider than the gap between versions, I’d add more.

Five, the gate. The new version must win or tie side-by-side. Names and numbers must be pronounced right, above a bar we agree with research. And time to first audio must stay inside the latency budget.

To close the gap, I’d spend my first weeks with whoever runs Bulbul evaluations now, listening to the failures myself.

I haven’t worked inside a training cycle. My experience is evaluating speech models, defining quality measures and deciding when something is ready to launch. That’s the part of the cycle I’d own from day one.

Here’s how I’d run an ASR release with research.

One, start with the problem in numbers. Which languages, accents and words fail, with real audio and a target. ‘Marathi account numbers wrong on noisy calls’ is a goal research can work on.

Two, agree the data. What to collect, from which sources, with consent, and the labelling guidelines. The test set stays separate from anything used for training.

Three, freeze the evaluation before training starts. The same test set and the same scoring rules for every checkpoint. I don’t move the goal mid-cycle.

Four, make the trade-offs. If a checkpoint gains accuracy but adds latency or cost, I make the call with research, based on customer impact.

Five, release. A gate, a staged rollout, and monitoring on live traffic.

From research I need what’s feasible, what the data and compute will cost, and early checkpoint results. From me they need clear priorities, real failing audio, a stable test set and fast decisions.

In my first month I’d learn how your training cycles run today.

I started as an analyst, at PayLane and then on returns at Shopwise. So I’m used to doing the analysis myself.

With 10,000 transcripts I’d go in four steps.

One, join each transcript to its reference and its metadata. Then check the metadata. Missing language tags or noise labels would skew every cut.

Two, normalise the text before scoring: case, punctuation, numbers, and Hinglish spelling and script. Otherwise I’d be measuring spelling.

Three, compute WER per file, then roll it up by language, accent and noise level. I’d weight by word count, so short clips don’t distort the averages.

Four, look at the spread as well as the average, and note how many files sit in each slice.

What I’d look at first: the slices with the most calls and the worst WER. That’s where customers feel it. Then I’d listen to the worst few dozen files in that slice. Listening tells me whether it’s noise, accent, names or a labelling mistake.

Saaras V4 already handles code-mixed speech, regional accents and noisy audio, with streaming under 150 milliseconds. So my first question would be scope. Which customers and use cases matter most this year? The ideas below are guesses I’d test against that.

One, the words that matter. My guess, from call-centre audio, is that names, numbers and amounts hurt customers most. A wrong digit in a loan amount breaks the call.

Two, regional languages on phone-line audio. At CallSense, Marathi calls failed before a bank launch. I’d check each language on real call audio, and rank them by customer volume and error rate.

Three, prove it offline. A fixed test set per language and condition, held out from training. Accuracy on names and numbers, plus WER and CER by slice. The new version has to win on the target slices and hold everywhere else.

Four, prove it live. A staged rollout to a few customers, with a business measure next to the model score. At CallSense that was usable transcripts. Here it might be how often an agent asks the caller to repeat.

Five, watch it after release. Monitor live traffic by language and customer, with a rollback trigger agreed in advance.

At CallSense, the usable-transcript measure I defined now sits in every customer’s weekly report. I’d want a number like that next to every model win here.

I’d hold it for that bank, and decide the wider release separately.

One, what MOS tells me. MOS measures how natural the voice sounds on average. A collections agent that says a loan amount wrong has failed, however natural it sounds. For banking, names and amounts need a gate of their own.

Two, measure the complaint. I’d build a test set from the bank’s own names and amount formats. Then I’d rate old and new versions on correct pronunciation. That shows whether it’s a regression or an old problem the bank now notices.

Three, decide. If the new version is worse on names and amounts, it stays out of banking until it’s fixed. If it’s no worse, the bank keeps its current setup while research fixes pronunciation. Other customers can move to the new version.

Four, release in stages. A small share of traffic first, watched by customer and language. A rollback trigger agreed in advance, tied to pronunciation errors and customer reports.

Five, keep checking. The pronunciation set runs on every version from now on, so this gets caught before release next time.

I made a similar call at CallSense. Auto-scoring for a bank was due to launch, and Marathi calls failed. I delayed the launch by three weeks, and we launched after the fix.

First I’d ask what matters most this quarter: which releases are coming, and which customers are waiting on them. My plan fits around that.

Days 1 to 30: learn. I’d go through the current test sets and metrics for Saaras and Bulbul. I’d sit with research to learn how a training cycle runs. And I’d hear customer complaints first-hand from sales and support, with the audio.

Days 31 to 60: one scorecard. For each model, results by language, accent and noise. For ASR, WER, CER and accuracy on names and numbers. For TTS, MOS, side-by-side results and pronunciation. Latency and cost for both. I’d agree the format with research so everyone reads the same numbers.

Days 61 to 90: use it on a release. I’d agree a release gate with research and apply it to one real release, with a staged rollout and a rollback trigger.

By day 90 you’d have three things from me. A scorecard the team trusts. A release gate used once for real. And a ranked roadmap for the next two quarters, built from customer pain and scorecard gaps.

At CallSense I defined the usable-transcript measure, and it’s now in every customer’s weekly report. That’s the kind of shared number I’d build here.

I’d go in four steps.

One, get the audio. I’d collect real calls from that customer with account numbers in them, across their agents and noise conditions. Then label each account number with the right digits.

Two, define the metric. Account-number accuracy: the share of account numbers where every digit is right. One wrong digit counts as a miss, because that’s how the customer experiences it.

Three, set a baseline and a target. Measure today’s model first. Agree the target with the customer, tied to how often their staff fix numbers by hand today. Then add guardrails: overall Marathi WER, and other languages, must not get worse.

Four, split the data. Keep a held-out test set from that customer. Give research separate examples for training, with notes on the failure patterns: digits spoken in English, in Marathi, or mixed.

I’ve done the first part of this at CallSense. When Marathi calls failed for a bank customer, I added 300 labelled Marathi calls to the test set.

That number uses the usable-transcript measure I defined: the share of calls where QA auditors didn’t need to listen to the audio.

It went from 61% to 84% after we switched vendors.

My part was two things. I designed and ran the three-vendor test that led to the switch. And I defined the measure the result is reported in.

A before-and-after number also moves with customer mix and call types. So I treat it as a strong sign the switch worked. The controlled evidence is the vendor test: three vendors on the same 2,000 labelled calls.

I joined Shopwise in June 2020 as a data analyst on returns. I moved into product in [month] 2021, as Associate Product Manager on search relevance. The CV shows that final title for my whole time there.

On search relevance I worked on how results were ranked. I ran A/B tests on search ranking.

That work is close to this role. You decide what good looks like, measure it, and test a change against it. I took the same habit into speech evaluation at CallSense.

I left in December 2022 to join a startup and own a product end to end.

I was at PayLane for nine months, from September 2019 to May 2020. I did analytics for their consumer lending product.

The company shut down its lending product, and my role went with it.

From there I joined Shopwise in June 2020 as a data analyst on returns, and moved into product the next year.

At CallSense we were about to launch auto-scoring for a bank customer. It scores their recorded support calls against their checklist. The launch date was set.

Then Marathi calls failed. I made the call to delay the launch by three weeks. Launching on time would have meant scoring the bank’s Marathi calls on transcripts we knew were failing.

I also added 300 labelled Marathi calls to the test set, so we could measure Marathi directly.

We launched the bank’s auto-scoring after the Marathi fix.

What I took from it: a launch gate has to cover every language the customer uses.

My notice period is 60 days, and my manager may release me in about 45.

I’ll relocate to Bengaluru by December. I told the recruiter I’m open to relocating, and that still stands.

If my start date lands before December, is starting before the move workable for you?