Auto-Captions in Video Translations: Helpful or Hurtful?

With all the forms of video media online, from long-form documentaries on platforms like YouTube and Amazon to short-form videos on Instagram Reels and TikTok, auto-captions are an addition that has had its fair share of ups and downs. YouTube is well known for its implementation of auto-generated captions back in 2009 after realising that encouraging creators to add them was not enough. However, how do auto-captions work, and why are they so borderline in their usefulness?

    To start with, we have to go back to the start of how auto-generated captions came about. The foundation of captions is related to the first automated speech recognition in computers. The software itself had a 42,000-word vocabulary in 1962 with IBM and translated speech to text in real-time.     Now, in the modern age, with all sorts of new language training models implemented to enhance speech recognition and with the help of AI, this software has a higher capability than it used to in the decades that have passed. 

William C. Dersh testing the experimental IBM speech recognition system

    Now that we know the basics of auto-captions, we can talk about how commonplace they are. Auto-captions apply to the majority of videos on YouTube, and other apps have also implemented their version. However, most auto-captions provided default to English, even though the video content online is worldwide and consists of multiple languages. 

An example of auto-generated captions on YouTube

On a similar note, the software itself is not good at detecting English words with accents. Such examples can be seen in videos in which the speakers are British, Irish, or have any audible accent that is not typically American. The inflexions and nuances in pronunciation in these accents are not picked up by the speech recognition software primarily because the software itself is not modelled properly.     Thanks to bias on a systemic level and developers training the software on exclusively American English examples, the software falls short and even wrongly generates captions. 

Popular Irish YouTuber Jacksepticeye doing a reaction video in which he says the word "thyme". He explains this as seen. The auto-captions pick it up as "time" instead. Funnily enough, in this video, Jack reacts to Irish accents that even he doesn't understand.

Nevertheless, just because auto-captions do not generate properly for some videos, it does not mean we should get rid of them altogether. Auto-captions still have benefited us since they make video content accessible for the deaf community and generally help us understand audio in content that is difficult to discern. In videos and content with overwhelming background noise or movies with loud fight scenes, auto-captions pick up dialogue and show it to the viewer in real-time, allowing us to understand the content despite those problems.     Such an experience is so prevalent that there are even memes made about how it feels next to impossible to comprehend dialogue and conversations in videos without captions. The video below by Vox is an example of why the issue occurs.

    So, are auto-captions helpful or hurtful? While there is no single decisive answer, it is clear that auto-captions have their benefits. While it is hateful in the sense of perpetrating bias, this can be corrected by training the software with more diverse information and languages to extend its possibilities and accuracy.