Benchmarking ASR Models on Brazilian Portuguese Accented Speech: a Case Study with Piauí Accent

Voice interaction with artificial intelligence systems has improved to the point that it is increasingly used as the primary interaction between people and those systems. However, this technology doesn't have equal performance across all language variants, such as regional or sociolinguistic dialects. The training of an automatic speech recognition model is heavily data-dependent, which places particular pressure on the selection of speech variants present in the training data. The effect of any variant in speech is currently measured by Word Error Rate (WER), the same metric that aggregates errors from an entire dataset. WER is a simple and practical metric, but at the expense of hiding model behavior that may be useful for analyzing speech variants. In this article, we propose a benchmark for evaluating accented speech, detailing data collection and the use of average WER, the 90th percentile of WER, and a qualitative analysis on a sample of the transcriptions to locate transcription errors. We demonstrate this approach by creating partitions of speech datasets based on the regional accent of Piauí and evaluating 11 state-of-the-art models on this regional accent. The results show that all models face unique difficulties in transcription due to elements of the regional accent, such as regional expressions and changes in the pronunciation of certain words. These elements are captured in the qualitative analysis. Analyzing the WER and comparing it to a dataset of read speech helps in the choice of models, and in the understanding of their limitations, while WER p90 presents itself as an interesting metric to understand the low end of their performance.