Computer Science 8803 DRL: Deep Reinforcement Learning

Course Logistics

Instructor: Animesh Garg (office: CODA S1145)
Teaching Assistants: Liquan Wang, Uzair Akbar, Albert Wilcox, Cherry Lian
Canvas will be used to take quizzes, view grades and view assignments.
EdStem: should be your first stop for questions and announcements.
Lecture: Tuesday / Thursday - 2:00 PM - 3:15 PM - Weber SST III, Room 2
Office Hours:

  • Animesh Garg: Tuesday 3:30 - 4:30 after lecture
  • TAs: TODO

Course Overview

Description

Robots of the future will need to operate autonomously in unstructured and unseen environments. It is imperative that these systems are built on intelligent and adaptive algorithms. Learning by interaction through reinforcement offers a natural mechanism to postulate these problems.

This graduate-level seminar course will cover topics and new research frontiers in reinforcement learning (RL). Planned topics include: Model-Based and Model-Free RL, Policy Search, Monte Carlo Tree Search, off-policy evaluation, temporal abstraction/hierarchical approaches, inverse reinforcement learning and imitation learning.

Learning objectives

At the end of this course, you will:

  • Acquire familiarity with state of the art in RL
  • Articulate limitations of current work, identify open frontiers, and scope research projects.
  • Constructively critique research papers, and deliver a tutorial style presentation.
  • Work on a research based project, implement & evaluate experimental results, and discuss future work in a project paper.

Textbooks and Resources

There is no official textbook for the class.

A number of the supporting readings will come from: Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition. This is available for free here and references will refer to the final pdf version available here.

Some other additional references that may be useful are listed below:

  1. Reinforcement Learning: State-of-the-Art, Marco Wiering & Martijn van Otterlo, Eds.
  2. Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig.
  3. Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

Additional Resources from similar courses.

  1. RL Course from UofT - Animesh Garg
  2. RL Course from UW - Byron Boots
  3. RL Course from Stanford - Emma Brunskill
  4. RL Course from University of Alberta - Martha White
  5. RL course at ASU/MIT - Dimitry Bertsekas
  6. David Silver’s course on Reinforcement Learning
  7. Deep RL Course from Berkeley - Sergey Levine

Prerequisites

You need to be comfortable with:

  • introductory machine learning concepts (CS 4644/7643/7641)
  • linear algebra
  • basic multivariable calculus
  • intro to probability You also need to have strong programming skills in Python.

Note: if you don’t meet all the prerequisites above please discuss with the instructor after class.

Optional, but recommended: experience with neural networks and introductory-level familiarity with reinforcement learning and control.

Grading

  • Homeworks: 20%
    • Two homework assignments
    • Programming + Short questions
  • Paper presentation and implementation: 30%
  • Quizzes and participation (live only): 10%
  • Project: 40%
    • Proposal: 5%
    • Intermediate Progress: 5%
    • Presentation: 10%
    • Final Report: 20%

Announcements

  • August 20 Welcome to Fall 2024!