SUMS 707 - Basic Reinforcement Learning
SUMS 707 Home Page
Basic Reinforcement Learning: Elementary Theory and Applications
Winter 2021
E-mails:
viet(dot)nguyen(at)mila(dot)quebec, moisescg(at)mila(dot)quebec,
shereen(dot)elaidi(at)mail(dot)mcgill(dot)ca, anna(dot)brandenberger(at)mail(dot)mcgill(dot)ca
Announcements
- Welcome to the first ever session of SUMS 707! This will be the course home page for the incoming semester. Suggested readings, assignments, course updates, and other useful
information shall be provided here. Course registration is currently open and managed by our TA, please email her if you have any related questions or
if you have any doubts regarding your background.
- The course will start in the beginning of the Winter 2021 semester and end a week before final exams start. Please see the course syllabus an/or schedule for a reference.
1.5 - 2 hour lectures will be delivered every week. The exact lecture dates are TBD (we currently have it on Mondays, but may change to accomodate students' busy winter schedules).
- The course will be taught on Zoom. Lectures might be recorded.
- This course is co-taught with Gabriela Moisescu-Pareja.
- Here is the lecture schedule.
- Here is the course syllabus.
- The recommended textbooks for the course are Reinforcement Learning: An Introduction by Andrew Barto and Richard S. Sutton,
Reinforcement Learning: Theory and Algorithms by Alekh Agarwal, Nan Jiang, Sham M. Kakade, and Wen Sun, Algorithms for Reinforcement Learning by Csaba Szepesvári,
Labelled Markov Processes by Prakash Panangaden, and Éléments de Géométrie Algébrique by Alexander Grothendieck, assisted by Jean Dieudonné.
Course Information
- Lecture Times: TBD - TBD
- Lecture Place: Zoom University Hall
- Office Hours: TBD by Zoom
- Office: Zoom
- TA and office hours:
-
Guest lecturer:
Recommended Background
- These are provided in the syllabus, but we give clarifications and provide emphasis here.
-
We expect a mildly adequate mastery of the following skill trees: (measure theoric) probstats,
linear algebra (even better if you know functional analysis), calculus (not really but we'll put it here),
machine learning (important, 1 course in ML + some experiences solving ML problems will be good enough for our purposes),
and category theory (very important).
-
Having learned the following dark spells will optimize your any% speedrun time: stochastics, martingale theory,
concentration inequalities, and higher topos theory.
-
Equipped with the following forbidden magics, you may see that the universe is more than a 10 dimensional blob of strings:
Fukaya category, Hitchin fibration, symplectic Lie n-algebroids.
-
You do not need forbidden magics. They are just cool topics of the discussions we occasionally have with Khoi (who will be
very excited to discuss them further with you).
-
But if you're self-driven enough to Google your way through the concepts you're struggling with, you'll probably be okay,
although we do clarify that the course is fast-paced.
- Category theory (very very important).
Course Materials
- For students who worry about prerequisites, please read and comprehend Chapters 1, 2, 3, 4, and 6 of Labelled Markov Processes.
You could try contacting the author himself for a pdf.
-
There are many good resources that cover the bases for linear algebra, calculus, ML,
you might find this book useful.
- Otherwise, nLab is where it's at.
- *cricket noises*
-
Lecture 1 Slides
-
Lecture 2 Slides
-
Lecture 3 Slides
-
Lecture 4 Slides
-
Lecture 5 Slides, recording on YT
-
Lecture 6 Slides
-
Lecture 7 Slides
-
Lecture 8 Slides
-
Exploration-Exploitation
in Reinforcement Learning. Lecture 8 continues with Part 3 and 4 of this tutorial.
Academic Integrity
Please do not cheat.
Every student has the right to submit written
work that is to be graded, in English or in French.
Chaque étudiant a le droit de soumettre en français ou en
anglais tout travail écrit.