Of Note Events and Neural Networks

12/22/2023

A research group at the University of Würzburg develops fundamental methods for analyzing music with the help of machine learning. The project is led by Professor Christof Weiß at CAIDAS.

Christof Weiß is working on the fundamental development of automated analysis methods for music recordings using data sets of classical music. (Image: Ingo Knopf / Klaus Tschira Stiftung)

In his role as Professor of Computer Science (Computational Humanities) at Julius-Maximilians-Universität Würzburg (JMU), Christof Weiß combines two apparently very different subject areas: Mathematics and computer science meet art and culture in his work.

As part of its Emmy Noether Programme, the German Research Foundation (DFG) is now funding his work with 1.4 million euros and enabling him to set up a new research group on “Computational analysis of music audio recordings: A cross-version approach”. The aim is to further develop automated analysis methods for music recordings, which could be used in computational musicology and the digital humanities in the future.

Music Data is Complex

"Analyzing music recordings is extremely difficult," explains Christof Weiß. "Music is multifaceted, characterized by different semantic dimensions such as time, pitch, timbre or style, which are highly entangled in the audio data. The presence of many correlated sources makes it more complex than spoken language, for example."

Furthermore, musical concepts are often ambiguous and subjective, which demands for interpretable methods and multiple annotations. However, it is precisely this annotated data that is often not available in sufficient quantities.

A Cross-Version Approach

This scarcity of data poses a particular challenge for deep-learning approaches, which therefore often hit a kind of "glass ceiling". Weiß and his team want to overcome this with a cross-version approach: "We want to use data sets of classical music that provide several modalities for each musical work, i.e. sheet music and recordings, several performances, and annotations by several experts."

Such datasets allow for transferring annotations between versions and for systematically evaluating the robustness of machine-learning method. Classical music offers a unique scenario for this.

The first step is to set up a framework with several datasets and develop evaluation strategies. These should enable a look behind the scenes of neural networks ("AIs") and algorithms: "Neural networks are almost always black boxes. We want to shed light on how they work and thus draw conclusions about the reliability of their predictions in music analysis," says Weiß.

Three Project Focuses

The research group, which will initially be funded for three years and can be extended for a further three years following a successful interim evaluation, will focus on three main areas of application. The first is dealing with fundamental representations of pitches and note onsets.

This is followed by local analysis methods, which focuses on aspects such as beats, downbeats, chords, and keys, and finally stylie analysis and categorization, which also involves similarities between different composers to be used for investigating questions of authorship.

Each of the three areas is linked to a PhD position, the first of which is to be filled in spring 2024.

Interdisciplinary Application Possibilities

The research group is part of the Center for Artificial Intelligence and Data Science (CAIDAS) and is located in the new Center for Philology and Digitality (ZPD). The researchers involved want to use the interdisciplinary expertise of both centers for their project work - an attractive and productive working environment.

"Our university is ideal for this type of research," emphasizes Christof Weiß. "In addition to computational musicology, the methods we develop can also be used in other areas of digital humanities and offer general insights into machine-learning methods."

About the Emmy Noether Programme

The DFG's Emmy Noether Programme is aimed at postdocs and temporary (junior) professors at an early stage of their academic career.

They are given the opportunity to qualify for a university professorship for life by independently leading a junior research group over a period of six years.