AI-driven content generation

Course goals

After the end of the course, the student will be able to:

• understand and explain the history and foundations of generative artificial intelligence
• understand and explain the mechanisms of visual generative AI (images, videos, 3D, animations)
• understand and explain the mechanisms of auditory generative AI (music, speech)
• understand and explain the mechanisms of linguistic generative AI (natural language, code)
• understand and explain how generative AI can be appointed to complex content environments (video games, interactive agents, multimedia)
• discuss ethical implications and issues connected to the rise of generative AI
• apply and execute contemporary approaches of generative AI
• engineer custom generative multimedia solutions to compound tasks

Assessment
The assessment is based on:

• a group project graded by technical implementation, documentation and presentation (55% of the final mark)
• weekly classroom assignments within hands-on sessions (5%)
• a final exam (40%)

To qualify for a repair of the final result the mark needs to be at least a 4, or “AANV”.

Prerequisites
The lecture is guided towards master students in Game and Media Technology and master students in  Artificial Intelligence.
Thus, the group project as well as classroom assignments require a basic level of programming experience.
If this is not available, students are encouraged to acquire this skill independent from this course and/or collaborate with proficient students in the course.

Content

This course captures the history, mechanisms and application cases of generative artificial intelligence towards a broader range of multimedia types.
Among others, generative AI has shown promising successes in producing images, videos, 3D objects, audio, text and compound media (such as video games). This can enable artistic expression for users that lack the artisanal skills for such tasks, accelerate industrial design and development processes and opens the door for novel, co-creative conceptions, but similarly involves numerous ethical implications.
In this course, we will not only outline how popular approaches of generative AI work, but also convey how to use these tools effectively, and discuss the potentials and hazards of this technology.

Course form
The course consists of two weekly sessions, from which usually the first one delivers theoretical knowledge about a particular type of multimedia and related approaches of generative AI, while the second session displays and deepens how to use these tools in practice.
In parallel, students are expected to work on a group project that integrates tools to produce complex multimedia content.

Literature

• Cao, Yihan, et al. "A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt." arXiv preprint arXiv:2303.04226 (2023) https://arxiv.org/abs/2303.04226
• Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. "High-resolution image synthesis with latent diffusion models"., 2022, in "Proceedings of the IEEE/CVF conference on computer vision and pattern recognition" (pp.10684-10695) https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf
• Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Rombach, R.,"Scaling rectified flow transformers for high-resolution image synthesis", 2024 https://arxiv.org/abs/2403.03206
• additional reading is to be announced in the lecture slides