Papersseum

Free course · no sign-in

Learn RL from zero

How machines learn by trying things, explained from the ground up. No machine learning needed. By the end you can train a bot of your own.

chapters
15
hours in total
11
machine learning needed
None
Start here · 20 minWelcome to RLWatch a clumsy agent learn a game by trial and error, and meet the loop that every RL problem shares.Start the course
Your progress0 of 15 opened
  1. A
  2. B
  3. C
  4. D
  5. E
Open a chapter and it fills in here.

Part A: Foundations

What reinforcement learning is, the little maths it leans on, and just enough Python.

3 chapters · 90 min
  1. 0020 minBeginnerWelcome to RLWatch a clumsy agent learn a game by trial and error, and meet the loop that every RL problem shares.
  2. 0140 minBeginnerThe maths toolkitSums, averages, expected value, slopes and a few friends. The only maths we need, rebuilt from the ground up.
  3. 0230 minBeginnerPython and NumPy, just enoughArrays, random numbers with seeds, and your first tiny experiment.

Part B: Choices and consequences

From one decision to a whole sequence of them, and how to plan when you know the rules.

4 chapters · 175 min
  1. 0345 minIntermediateBandits: try the unknown or stick with what works?Five slot machines and a hundred coins. Your first learning agent, and the exploration versus exploitation trade-off.
  2. 0440 minIntermediateMarkov decision processesStates, actions, rewards and why the future is worth a little less than now.
  3. 0545 minIntermediateValue functions and the Bellman equationHow good is it to be here? Answered with a small system of equations you can solve by hand.
  4. 0645 minIntermediateDynamic programming: planning when you know the rulesPolicy evaluation, policy iteration and value iteration, with a grid world you can poke at.

Part C: Learning from experience

What to do when you do not know the rules and can only learn by playing.

3 chapters · 140 min
  1. 0740 minIntermediateMonte Carlo: learn from whole gamesPlay a lot, average what happened, get better.
  2. 0850 minIntermediateTemporal-difference learning and Q-learningDo not wait until the end to learn. The idea behind most of modern RL.
  3. 0950 minIntermediateWhen tables get too bigWhy we need to guess about states we have never seen, and gradient descent built up from a single variable.

Part D: Deep reinforcement learning

Replace the table with a neural network, and learn what to do directly.

3 chapters · 165 min
  1. 1060 minAdvancedNeural networks from scratchA neuron is a weighted sum. Stack them, measure the error, and let the chain rule do the learning.
  2. 1150 minAdvancedDeep Q-networksA network in place of the Q-table, and the two tricks that stop it from falling apart.
  3. 1255 minAdvancedPolicy gradientsDo not learn values at all. Learn what to do, and nudge it toward what worked.

Part E: Many agents

What changes when the other players are learning too, and how to put it all to work.

2 chapters · 105 min
  1. 1345 minAdvancedMulti-agent RL and self-playA moving target, learning by playing yourself, and how rating systems like Elo work.
  2. 1460 minAdvancedBridge: build a Papersseum botThe game as an RL problem, a training recipe, and how to pack your policy into one file and submit it.

Reference

Keep these open in a tab while you read.