A live, self-correcting speech-to-text pipeline for classes taught in three languages at once — built on one laptop, by a broadcast engineer.
I attend live online classes on Vedanta and Sanskrit grammar. The teacher speaks Portuguese, quotes English, and recites Sanskrit — often inside the same sentence. I am a participant, not the host, so no platform-side transcript is available to me.
Running the audio through a general speech model produces something worse than useless: fluent Portuguese text in which every technical term is wrong. The model has no reason to know अमानित्वम्, so it emits the nearest Portuguese-sounding words and does it confidently. The words the transcript exists for are precisely the ones it destroys.
That is a generic problem wearing an unusual costume. Any domain with its own vocabulary — surgical, legal, liturgical, a broadcast rundown full of product names — breaks the same way, for the same reason.
Latency and accuracy pull in opposite directions, so I stopped trying to satisfy both with one pass. The recorder writes short blocks; a live path transcribes them as they land, and a post-session path re-transcribes the whole class as a single audio stream, which is measurably better because the model sees full sentences instead of arbitrary 20-second cuts.
The recorder and the transcriber are separate processes on purpose. This is ordinary broadcast discipline applied to software: never let a downstream device take the source down. If the transcriber stalls or the GPU locks up, ffmpeg keeps writing audio and nothing is lost — the text can always be rebuilt from the blocks, but the class only happens once.
Whisper does not emit pratipaksha-bhavana as one token sequence. It emits three or four ordinary Portuguese words that happen to sound like it. A naive find-and-replace table never matches, because the thing you are searching for was never written.
FixMatch on n-grams — join up to four consecutive output words before comparing against the lexicon. Substitutions are keyed on sound, not spelling.
This is the part that makes the system more than a dictionary. Classes open and close with fixed mantras, and the teacher cites chapter and verse before commenting on them. Those source texts are published and available. So the transcript is not unverifiable — a large part of it can be checked against the original wording.
FixA correction pass fetches the cited source, aligns the transcribed span against it, and rewrites what drifted. A review pass over the back catalogue found and corrected seven confirmed errors, each documented with its evidence.
Terms were being corrected in the text but silently dropped from the audio-clip extractor. The cause took a while to find: the correction table mapped a misheard sound to a term that was not present in the lexicon, so the downstream stage discarded it as unknown. Two components disagreed about what counted as a valid word.
FixEvery substitution target must exist in the lexicon — enforced, not assumed. The lexicon is now the single authority both stages read from: 327 terms with Devanagari, IAST transliteration and a Portuguese gloss.
Running the clip extractor while the live transcriber was still going locked up the Apple Silicon GPU and took both down. Found the expensive way, mid-session.
FixInference is serialized by design; the extractor only starts after the transcriber has stopped. Written into the operating procedure so a future me cannot get it wrong.
A transcript nobody can audit is a liability. Because the single-pass transcription carries word-level timestamps, the pipeline cuts a short audio clip for every technical term it claims to have found, and publishes them next to the term in a glossary. To check whether the system heard शौचम् or invented it, you press play.
| Devanagari | IAST | What the pipeline resolves it from |
|---|---|---|
| अमानित्वम् | amānitvam | Cited verse; matched against the published source text |
| शौचम् | śaucham | Lexicon entry plus n-gram match on the fragmented output |
| षड्रिपु | ṣaḍripu | Corrected after review; the enumeration is fixed and checkable |
| स्थैर्यम् | sthairyam | Chanted cadence flags the span as Sanskrit, not Portuguese |
Accuracy is currently estimated from spot-checking, not measured. There is no hand-corrected reference transcript yet, so there is no word error rate to quote — and I would rather say that than publish a number I cannot defend. Building that reference set and reporting WER per language is the next piece of work, and it is the difference between a system that looks right and one that is known to be.
Two failures were worth more than the features they cost.
The microphone heard itself. During one session the capture picked up my own playback of the glossary clips, contaminating the recording. The session was discarded. The lesson is the boring one every live engineer already knows and software people relearn: know exactly what your input is routed to, and check it before you roll.
A restart erased the live text. Splitting a long class into two sessions wiped the in-progress transcript, because the startup script cleared the live file without saving it. The startup script now archives the previous session before zeroing anything. Startup is not a neutral act; it destroys state unless you tell it not to.
Strip the subject away and the problem is: real-time speech, mixed languages, a closed vocabulary that general models get wrong, audio that cannot leave the building, and an output somebody has to be able to trust.
That is live captioning for a conference with product names in it. It is a medical or legal proceeding where the terminology is the content. It is a multilingual corporate event where the client wants subtitles on the programme feed and will not accept a cloud service touching the audio.
I come at these from the signal side. My working life is live event and broadcast production — vMix, NDI, SRT, SDI infrastructure, FFmpeg — so the transcription is not a demo running on a laptop in isolation: the audio arrives as a real source from a real switcher, and the captions have to come back as something a production can put on air, on time, every time. Most people who can build the model layer have never patched an audio feed under time pressure. Most people who can do that have never built the model layer.
The pipeline is heavily AI-assisted at the implementation level. What is mine is the part that decides whether the output can be trusted: framing the problem, choosing to split the live and post-session paths, designing the correction layer against primary sources, the n-gram matching strategy, the decision to publish an audio clip beside every term, and the judgement that an unmeasured accuracy claim does not go on this page.
The idea. The rules the build has to follow — programming and commit discipline, the test harness, the security model, and how the product is organised for market. Choosing the technology. Coordination, and the instructions that direct the work. Testing against real signal, real audio and real hardware, and the fine tuning until it behaves. Deciding what ships and what gets rolled back.
The implementation itself — the code, the platform plumbing, the first drafts of documentation.
I disclose it because hiding it would misrepresent how the work happens now, and because the skill that actually matters here is not typing the code. A model will happily produce a transcription pipeline that looks finished and quietly drops every term it does not recognise — which is precisely the bug described above, and it took real debugging to find. Knowing that the output must be checkable, and building the checking, is the engineering.
I do not write every line. I do read every line I integrate, and I keep the operating rules written down where the next session will read them. Where I am still working: measuring instead of estimating, and closing the gap between what I can review and what I could write unaided.
Broadcast and live-event engineer, São Paulo, Brazil. Partner at VENG (Vídeo Engenharia), working in professional AV, broadcast technology and technical training. I build AI systems for production environments where the audio and video are real, the deadline is the doors opening, and the vocabulary is specific.
AV and broadcast operations that want AI in the workflow and need someone who understands both halves.
Live captioning, transcription and production automation, delivered through VENG.
Applied AI for AV and broadcast professionals — in Portuguese, English or Spanish.
Remote technical collaboration with teams outside Brazil — contract projects, partnerships and joint work across time zones.
joao@video.eng.br · +55 11 99602-1111