Reinforcement and Causal Learning

Course goals

After completing the course, the student

• knows the main concepts and algorithms of reinforcement learning
• can describe the role played by function approximation methods (such as neural networks) in reinforcement learning
• is able to explain the differences among different types of RL methods (e.g. on-policy VS off-policy, control VS prediction)
• understands the description of causal relations in terms of directed graphs and structural equation models
• can explain the implications of causal assumptions
• is able to test these implications using observational and interventional data
• can implement machine learning methods in such a way that causal information is taken into account

Assessment 

• two written exams (each 35% of the final grade)
• programming assignments with written reports (30%)

Theoretical exercises (ungraded) will be discussed during tutorial sessions as preparation for the exam.
In order to pass the course, your exam grade must be at least a 5.0.

To qualify for a repair of the final result the mark needs to be at least a 4, or “AANV”.

Prerequisites
The following knowledge will be assumed in this course:
-  solid proficiency in mathematics, in particular probability theory (e.g. ability to understand and manipulate formulas involving conditional probabilities and expectations), linear algebra, basic calculus
-  programming skills in Python
-  understanding of basic machine learning theory and methods, for example from bachelor courses in Machine Learning

Content

Modern machine learning methods have achieved spectacular results on various tasks. Yet there are pitfalls and limitations that can't be overcome simply by increasing the amounts of data and computing power.
For example, standard methods assume that the data are drawn from a single, unchanging probability distribution.
The two main topics that we cover in this course both deal with situations where that is not the case.
First, reinforcement learning is about the design of agents that can learn to interact with an unknown environment. After studying the basic concepts of reinforcement learning, we will go on to see how reinforcement learning methods can be improved by building on recent advances in supervised learning (such as deep learning). This brings with it a unique set of challenges that we will cover in this course.
The second topic, causal inference, is the subfield of machine learning that studies causes and effects: if we make a change to one random variable in a system, for which other variables does the distribution change? An understanding of these cause-and-effect relations allows us to predict the results of a change in the environment. We will also look at the problem of learning these relations from data.

Course form
Lectures, tutorials/practical sessions