Sound and music technology

Course goals

During the course you will acquire basic knowledge and understanding of different aspects of musical information, such as melodic, rhythmic, harmonic and timbral information.
At the end of the course you will be able to address the specific challenges involved in modelling these aspects of music information for key topics in sound and media technology.
You will be able to evaluate the pros and cons of different state-of-the-art techniques developed for specific important tasks in games and media.
Moreover, you will be able to reflect on the potential of selected computational methods for modelling specific aspects of sound and music in the games and media context, to discuss perspectives of improvements in current computational approaches, and to apply your knowledge about these techniques in a creative manner within a group project.
After module A you will be able to:

• use appropriate terminology for describing sound and music through features
• distinguish and recognize different ways sound and music are used in games
• assess and discuss emotional and affective qualities of sound and music and their role for games

After module B you will be able to:

• understand differences in basic and complex feature extraction and processing in audio and symbolic formats
• understand and discuss different ways of evaluating computational models for classification and retrieval of musical objects based on features
• creatively apply existing toolboxes for sound and music analysis
• implement selected algorithms from the literature and have ideas on improvement

After module C you will be able to:

• address specific challenges for creating sound and music for games
• understand and utilize different forms of manipulation in audio and symbolic formats
• assess the potential and challenges of different computational models for automatic music generation (such as composition, accompaniment, improvisation), including considerations of employing AI in creative contexts

Assessment
During the course we will assess your learning progress through short questionnaires and discussions during class, and through the application of course content by students to stimulate exam preparation with your classmates in short presentations.
At the end of the course, we will assess the learning outcomes of the course in a written exam, and the application of the course content in the final group project.
The final grades will be determined as follows, while the exam needs to be passed in order to pass the course:

• final project (45% of the final mark)
• exam (55%)

To qualify for the retake exam, the final grade needs to be at least a 4 (or “AANV”).

Content

Sound and music provide powerful ways of impacting the human experience involved in the engagement with games and media.

For instance, they are of crucial importance for the interactive and immersive qualities of games, which are key aspects in making games worthwhile to play.

Sound and music are also crucial in the context of media technology. In order to enhance users’ experiences when listening to, interacting with and searching for music from large collections of digitized music that became available over recent decades, new technologies for processing digitized music and sound information need to be developed.
In this course, you will learn how to apply and develop computational methods to extract, process and utilize music information from digital sound and music in the context of newly emerging research areas within games and media.
You will learn how sound and music information is crucial for the human experience, and how the computational modelling of sound and music contributes to the enrichment of this experience in games and media.
This means that you will get to know both basic concepts on how human listeners extract, make sense of and give meaning to information from sound and music, and how these basic concepts are used, researched and applied through computational technology.
The course is structured around three main modules:

1. sound and music for games
2. analysis, classification, and retrieval of sound and music for media
3. generation and manipulation of sound and music for games and media

The course will cover key topics for sound and music technology in the context of games and media, such as interactivity and immersion in games through sound and music (A), classification and retrieval of similar musical objects in multimedia (B), and the utilization of the emotional and affective qualities of music in games and media (A, C).
You will learn what specific technologies are developed and required within these key topics, such as automatic pattern discovery, voice separation, and feature extraction and manipulation (B).
For studying, discussing and employing these technologies you will get to know different representation forms of music information in audio and symbolic data (A, B), different musical dimensions such as melody, rhythm, harmony, timbre and loudness (A, B), and how they are modelled through computational features (A, B, C).
Moreover, you will learn about different general strategies for developing computational models for sound and music processing, such as model-based versus data-driven approaches, and about the challenges of evaluating these models.

Course form
Lectures that cover basic terminology, models and techniques; application of course content by students in short presentations, aimed at stimulating exam preparation with classmates; learning by doing through a final project.
Students will present their project in the final week.