All learning paths Learn See Try Make Handle mixed script and spelling What this lesson is about People write one language in two scripts and spell it four ways. Decide which of those you keep.
A chat set has lines in two scripts, and one word is typed two ways. The batch is due tomorrow. What now?
A Keep the lines as they are, and fix only the spellings that look like typing slips. B Ask the people who typed the lines which spelling they meant, and use that. C Decide what you keep as written and what you make standard.
Check my recall Why this matters IndicVoices keeps two layers: one written exactly as spoken, one standardised to the textbook form. Two layers let a later team choose, which one flattened layer never can.
Decide which spellings and scripts you keep, which you standardise, and write the change log.
IndicVoices keeps two layers: one written exactly as spoken, one standardised to the textbook form. Two layers let a later team choose, which one flattened layer never can.
Decide what you keep as written and what you make standard. Standardising too hard destroys the very thing the collection was made for. Record every change you made, so somebody can undo it later. Task A Hinglish chat set: some lines in Devanagari, some in Latin letters, one word often typed two ways.
Weak approach Everything is converted to Devanagari and every spelling is corrected to the textbook form.
Stronger approach The original line is kept. A second column carries the standard form. A note says which spellings were changed.
Why the stronger approach works The first version cannot answer the question the collection was built for: how do people really write? That is the second rule.
Try a changed situation You are now cleaning speech clips, and a tool can strip the market noise out of every one of them. The clips are for a model that must work in markets.
Make something yourself Take one mixed-script sample of twenty lines. Write what you kept, what you made standard, and a change log a stranger could use to undo your edits.
Lesson 6 of 12 on Bharat-language data craft next Record speech worth keeping Part 2 ends in a work sample you can send The rules this work sample is read against
Read the primary or official source: Javed et al., IndicVoices (arXiv)