ASR for AI Development: Benchmarking Technical Terms Recognition
How to measure speech recognition quality in voice-driven AI development. Comparing GigaAM, T-one, and Whisper models and the flaws of WER.

Since last November, the author has built a Dark Software Factory, where the majority of development is performed entirely using voice commands. What started as a simple experiment quickly evolved into a streamlined workflow for product creation on the go. The foundation of any such automated software factory is ASR (Automatic Speech Recognition), as the accuracy of converting voice to text directly dictates the pipeline's speed.
- A benchmark was created featuring 1000 voice recordings (20.3 audio hours, 7 speakers) focusing on PDLC tasks.
- 454 unique technical terms were identified from 18,000 word forms and meticulously annotated in the audio.
- All models (GigaAM, T-one, and Whisper) were tested exclusively on a CPU (AMD Ryzen 5 3600) without GPU acceleration.
- The classic WER metric was proven tricky for development, as errors in short critical terms can easily hide in overall stats.
Dataset Features and Testing Methodology
To conduct an objective test, a dataset combining the author's personal audio and technical public-source podcasts was compiled. The speech features a complex mixture of conversational Russian, numerous Anglicisms, professional slang, and borrowings. Out of 18,000 unique word forms, 454 specific technical terms related to interfaces, architecture, and frameworks were carefully isolated and annotated.
The benchmark ran on a standard machine equipped with an AMD Ryzen 5 3600 CPU (6 physical cores) and 63 GB of RAM, running entirely on the CPU without GPU usage. The primary contenders evaluated were GigaAM, T-one, and Whisper. The evaluation framework utilized standard metrics including Word Error Rate (WER), Character Error Rate (CER), Real-Time Factor (RTF), and exact term match percentages.
Read alsoNeuralink Blindsight: Elon Musk details vision-restoring neural chipModel Comparison Results and the WER Dilemma
Speed tests showed that GigaAM and T-one are true CPU champions, while Whisper struggles without hardware accelerators. GigaAM emerged as the winner in pure WER, proving its excellence in general Russian speech transcription. However, when examining specific technical terms, Whisper demonstrated a stronger capability in capturing those entities, revealing a clear trade-off between general text accuracy and technical term precision.
The article deeply examines the Word Error Rate (WER) metric, which counts substitutions, deletions, and insertions needed to match the ground truth. An identical low WER score can hide either a successful execution or a ruined feature if the error occurs in a critical technical term. In long audio recordings, errors in rare terms have a small impact on the overall numerator and denominator, yet they can severely misdirect an AI coding agent.
Practical Implications for Software Factories
In practice, the generated transcript feeds into a harness that splits it into user intent and code search tasks. While search routines can tolerate minor semantic variations (like dashboards and synonyms), misrecognizing a rare technical protocol or library can completely derail the entire task. Amusing failures like converting precision into a completely unrelated word or mistaking production environments can cause severe issues in automated commits.
For developers and tech businesses exploring voice interfaces and AI-driven workflows, this research highlights that general-purpose speech recognition models are not enough. Building reliable voice-controlled development pipelines requires domain-specific benchmarking, careful handling of technical jargon, and robust post-processing layers to filter out ASR transcription noise.
Source: Все публикации подряд на Хабре



