FIAT/IFTA World Conference 2024

In the audiovisual archive of the Bayerischer Rundfunk, speaker diarization systems are used to extract metadata from audio content. These systems can automatically annotate who spoke when in an audio stream, but yet their performance does not match the one of an archivist given the same task. That calls for evaluation. Therefore, as the final project of my traineeship, I have developed an application for testing the speaker diarization systems used in the Bayerischer Rundfunk, introducing scores and error rates.

As of yet, an interactive prototype has been built, showcasing a user interface for the application. Furthermore, it has been laid down how to access the speaker diarization systems via API, and several Python scripts have been written for converting mining results as well as importing metrics from an open-source benchmarking toolkit. This toolkit provides a set of classes to compare the output of speaker diarization systems to perfectly annotated reference data. Finally, a series of tests has been conducted which has proven the concept to be applicable for reproducible testing.

Using the application, single versions of speaker diarization systems as well as different speaker diarization systems can be compared quality-wise. This will make it easier to decide whether to implement said services, and secure the audiovisual archive's data quality.