Chapter 1
What Is Robot Learning?
Textbook: Sutton & Barto §1.1, 1.3, 3.1, 17.3–17.4
Reinforcement learning is learning what to do—how to map situations to actions—so as to maximize a numerical reward signal. The learner is not told which actions to take, but instead must discover which actions yield the most reward by trying them.
The perception–action loop
A robot reads the world through its sensors (perception), decides what to do (decision), and changes the world with its motors (action). The changed world comes back in through the sensors. Sutton and Barto (§3.1) describe this loop as an interaction between an agent and an environment. At each time step \(t\) the agent receives the environment’s state \(S_t\) and selects an action \(A_t\). One step later the environment returns a numerical reward \(R_{t+1}\) and a new state \(S_{t+1}\). The interaction is therefore recorded as a trajectory \(S_0, A_0, R_1, S_1, A_1, R_2, \dots\). The book notes that it uses the words agent, environment, and action in place of the engineers’ controller, controlled system (plant), and control signal.
Classical control and learning
In classical robotics, engineers build a model of the robot and its environment (equations of motion, maps, the shapes of objects) and design controllers and path planners on top of it. On a factory line where the model is accurate, this approach is fast and dependable. But where a model is hard to write down, such as cloth that changes shape, objects the robot has never seen, or work done alongside people, rules are hard to write by hand. Learning obtains the rule for acting, the policy, from data and experience instead. The price is that it needs a lot of data, that it is hard to explain why a learned behavior is the way it is, and that it can fail in situations it has never seen.
What reinforcement learning is made of
The defining features of reinforcement learning are trial and error and delayed reward, because an action affects not only the immediate reward but also the next situation and all the rewards after it. Another is the exploration–exploitation dilemma. To collect a lot of reward the agent must choose actions that worked before, but to find better ones it must also try actions it has not tried (§1.1). The book (§1.3) names four elements of a reinforcement learning system.
- Policy: the rule that maps states to actions. It alone determines behavior.
- Reward signal: a single number the environment sends at each time step. It defines the goal of the problem.
- Value function: the expected total reward to be collected from a state onward. If reward is what is good right now, value is what is good in the long run.
- Model of the environment (optional): given a state and an action, it predicts the next state and reward. Methods that plan ahead with a model are model-based; methods that learn by trial and error alone are model-free.
A map of robot learning
If you do not know what the reward should be but can watch someone who is good at the task, you can learn from demonstrations instead. The book (§17.4) calls this imitation learning or learning from demonstration. This guide is divided into four parts by what the robot learns from.
| Learning signal | What the policy learns | In this guide | Representative methods |
|---|---|---|---|
| Reward | Actions that make the total reward as large as possible | Ch. 1–6 | Value iteration, Q-learning, PPO |
| Demonstrations | What an expert did in each situation | Ch. 7–8 | Behavior cloning, DAgger, ACT, Diffusion Policy |
| Large-scale robot data and language | A policy that works across many robots and tasks | Ch. 9–11 | RT-2, OpenVLA, OpenVLA-OFT, π0 |
| Human comparisons | Qualities that are hard to define in words | Ch. 12–13 | RLHF, DPO, FPL |
Drive a gridworld robot yourself
Move the robot to the goal G with the arrow buttons or the arrow keys. Every step gives a reward of −1, reaching the goal gives +10, and falling into a pit (✕) gives −10; both end the episode. The log table collects the \(S_t, A_t, R_{t+1}\) that the agent and the environment exchange.
| t | State St | Action At | Reward Rt+1 |
|---|
With slip turned on, the same action does not always have the same result. A good policy must therefore think about the probabilities of outcomes, not just “the next cell” (chapter 2). With the robot’s view turned on, the robot sees only the cells next to it. Sensors that reveal only part of the state are something real robots live with all the time (Sutton & Barto §17.3).
Chapter 2
MDPs and Rewards
Textbook: Sutton & Barto §3.1–3.5
A Markov decision process (MDP) is a mathematical framing of the problem of achieving a goal through interaction. It consists of a set of states \(\mathcal{S}\), a set of actions \(\mathcal{A}\), a set of rewards \(\mathcal{R}\), and the environment’s dynamics \(p\).
\[p(s',r\mid s,a)=\Pr\{S_t=s',\,R_t=r\mid S_{t-1}=s,\,A_{t-1}=a\}\]The probability that taking action \(a\) in state \(s\) leads to next state \(s'\) with reward \(r\) (Sutton & Barto eq. 3.2). In experiment 1-1, turning on slip splits this probability across several outcomes.
The Markov property says that the next state and reward depend only on the immediately preceding state and action, not on any earlier history. The book suggests viewing this as a requirement on the state rather than on the decision process: the state must contain every piece of past information that matters for the future. Just as you cannot predict the next moment from a pole’s angle without its angular velocity, a robot’s state usually has to include velocities along with positions.
The reward hypothesis
The book (§3.2) states the starting point of reinforcement learning like this: “all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward).” The reward is how you tell the robot what you want, not the place for instructions on how to do it. Reward a chess agent for every piece it captures and it may find ways to capture pieces even at the cost of losing the game. Chapter 6 shows this trap firsthand.
Returns and discounting
What the agent maximizes is not the immediate reward but the sum of rewards to come, the return \(G_t\). An interaction with a natural end, such as one game or one run through a maze, is an episode. In tasks without an end the sum could grow without bound, so future rewards are reduced by a discount rate \(\gamma\) (\(0\le\gamma\le 1\)) before they are added.
\[\begin{aligned}G_t&=R_{t+1}+\gamma R_{t+2}+\gamma^2R_{t+3}+\cdots=\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}\\ G_t&=R_{t+1}+\gamma\,G_{t+1}\end{aligned}\]A reward received \(k\) steps later counts \(\gamma^k\) times as much as one received now (eqs. 3.8–3.9). If every reward is +1, \(G_t=1/(1-\gamma)\) (eq. 3.10).
With \(\gamma=0\) the agent is myopic and looks only at the next reward; the closer \(\gamma\) is to 1, the farther ahead it looks. The recursion on the second line leads to the Bellman equation of chapter 3. Finally, a policy \(\pi(a\mid s)\) is the probability of choosing action \(a\) in state \(s\) (§3.5). Reinforcement learning is a way of changing this policy with experience.
A small reward nearby or a big one far away
The robot picks one of two routes. Route A ends at a small charging station after 3 steps; route B ends at the goal after 10 steps. The outline of each bar is the reward received, and the filled part is what is left after discounting, \(\gamma^k R\). The agent picks the route with the larger return.
Chapter 3
Value Functions and the Bellman Equation
Textbook: Sutton & Barto §3.5–3.6, 4.1–4.4, 4.6
Under a policy \(\pi\), the value \(v_\pi(s)\) of a state \(s\) is the expected return when starting from \(s\). The action value \(q_\pi(s,a)\) of a state–action pair is the expected return when taking \(a\) in \(s\) first and following \(\pi\) afterward (eqs. 3.12–3.13). Thanks to the recursion \(G_t=R_{t+1}+\gamma G_{t+1}\), a state’s value is tied to the values of its neighbors.
\[v_\pi(s)=\sum_a \pi(a\mid s)\sum_{s',r}p(s',r\mid s,a)\big[r+\gamma\,v_\pi(s')\big]\]The Bellman equation (eq. 3.14). The value of a state equals the next reward plus the discounted value of the next state, averaged over how likely each is.
A policy whose value is at least as large as any other policy’s in every state is an optimal policy \(\pi_*\), and its value function is \(v_*\). The optimal value satisfies a self-consistency condition that needs no particular policy, the Bellman optimality equation. Once you know the optimal action value \(q_*\), simply choosing the action with the largest \(q_*\) in each state is optimal behavior.
\[\begin{aligned}v_*(s)&=\max_a\sum_{s',r}p(s',r\mid s,a)\big[r+\gamma\,v_*(s')\big]\\ q_*(s,a)&=\sum_{s',r}p(s',r\mid s,a)\Big[r+\gamma\max_{a'}q_*(s',a')\Big]\end{aligned}\]The Bellman optimality equations (eqs. 3.19–3.20). A maximum replaces the average over the policy.
Dynamic programming: when the model is known
If you know the environment’s dynamics \(p\), you can turn the Bellman equation directly into an update rule and compute values. This is dynamic programming (DP) (book chapter 4).
- Policy evaluation: repeatedly updating every state once with the right-hand side of eq. 3.14 (one sweep) converges to \(v_\pi\) (eq. 4.5).
- Policy improvement: acting greedily with respect to \(v_\pi\), choosing the action that looks best one step ahead, gives a better (or at least no worse) policy.
- Policy iteration: alternating evaluation and improvement makes the policy improve monotonically, and in a finite MDP it reaches an optimal policy in finitely many iterations (§4.3).
- Value iteration: stops evaluation after a single sweep and takes the maximum right away. It is the Bellman optimality equation turned into an update rule (eq. 4.10).
The book calls this picture, in which evaluation and improvement pull each other toward the optimum, generalized policy iteration (GPI) (§4.6). Most of the methods from chapter 4 of this guide on fit inside it. DP, however, needs to know \(p\) and has to sweep every state, so it cannot be used as is for robot problems with enormous numbers of states.
\[v_{k+1}(s)=\max_a\sum_{s',r}p(s',r\mid s,a)\big[r+\gamma\,v_k(s')\big]\]The value iteration update (eq. 4.10). From any \(v_0\), it converges to \(v_*\).
Value iteration and policy iteration, one step at a time
The number in each cell is the state value computed so far, and the arrow is the greedy policy for that value. Press “One step” and watch the values spread outward from the goal. On the robot map you can tap an empty cell to place a wall or a pit.
Chapter 4
Q-Learning and Exploration
Textbook and papers: Sutton & Barto §2.2, 6.1, 6.4–6.5 · Watkins (1989)
The dynamic programming of chapter 3 needed the environment’s dynamics \(p\). A real robot usually does not know that model. It has to learn instead from the experience \((S_t, A_t, R_{t+1}, S_{t+1})\) it gathers by actually moving.
Temporal-difference learning
Monte Carlo methods wait until the episode ends and the actual return \(G_t\) is known before they correct a value. Temporal-difference (TD) learning waits only one step. It forms a target from the next reward and the current estimate for the next state and corrects right away (§6.1). Because it updates an estimate from another estimate, it is said to bootstrap, as DP does. The difference in the brackets, \(\delta_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)\), is the TD error.
\[V(S_t)\leftarrow V(S_t)+\alpha\big[R_{t+1}+\gamma V(S_{t+1})-V(S_t)\big]\]The TD(0) update (eq. 6.2). The step size \(\alpha\) sets how much to correct at once.
SARSA and Q-learning
To choose actions, action values \(Q(s,a)\) are more convenient than state values. There are two ways to apply the same TD idea to \(Q\).
\[\begin{aligned}\text{SARSA:}\quad Q(S_t,A_t)&\leftarrow Q(S_t,A_t)+\alpha\big[R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)\big]\\ \text{Q-learning:}\quad Q(S_t,A_t)&\leftarrow Q(S_t,A_t)+\alpha\big[R_{t+1}+\gamma\max_a Q(S_{t+1},a)-Q(S_t,A_t)\big]\end{aligned}\]Eqs. 6.7 and 6.8. The name SARSA comes from the five quantities the update uses, \((S_t,A_t,R_{t+1},S_{t+1},A_{t+1})\).
SARSA uses the value of the action \(A_{t+1}\) it actually chooses next. It learns the value of the policy it is following, so it is an on-policy method. Q-learning (Watkins 1989) uses the value of the best action in the next state. Whatever the agent actually did, \(Q\) directly approximates the optimal action value \(q_*\), so it is an off-policy method. The behavior policy still has to keep visiting every state–action pair so that all of them keep being updated (§6.5).
ε-greedy exploration
If the agent always picks the action with the largest estimate, it may never find a better action it has not yet tried. An ε-greedy policy acts greedily most of the time but, with a small probability \(\varepsilon\), picks one of all the actions at random (§2.2). It is the simplest answer to the exploration–exploitation dilemma of chapter 1.
Cliff walking
The book’s example 6.6 shows how the two methods differ. A 4×12 grid has a cliff along its bottom edge, with the start S and the goal G at its two ends. Every move gives −1, but stepping into the cliff gives −100 and sends the agent back to the start. Exploring with \(\varepsilon=0.1\), Q-learning learns the values of the optimal path right along the cliff edge but occasionally falls off because of its random actions. SARSA takes that randomness into account and learns a safer path around the top, and the reward it collects while learning is actually larger. If \(\varepsilon\) is gradually reduced, both methods converge to the optimal path.
SARSA and Q-learning on cliff walking
Each algorithm is trained 20 separate times with the same settings, and the reward collected in each episode is averaged. The grid shows the greedy path of the first run of each. Changing ε restarts learning from scratch.
Chapter 5
Deep Reinforcement Learning
Textbook and papers: Sutton & Barto §13.1–13.5, 16.5 · Mnih et al. (2015) · Schulman et al. (2017)
In a gridworld we stored \(Q\) in a table with one entry per state. When the state is a camera image, the number of possible states is practically infinite and no table will do. Instead, a function \(\hat q(s,a,\mathbf{w})\) with parameters \(\mathbf{w}\) approximates the values. When that function is a deep neural network, we speak of deep reinforcement learning. Neural networks and back-propagation are covered in chapters 9–11 of the Machine Learning and Deep Learning guide.
DQN: Q-learning from the screen
The DQN of Mnih et al. (2015) learned 49 Atari 2600 games with the same network architecture and the same settings (book §16.5). The input is the game screen shrunk to 84×84 with the last four frames stacked, and the output is a \(\hat q\) value for each action. It learned from 50 million frames per game, about 38 days of experience, and reached at least 75% of a human tester’s score on 29 of 46 games. These devices stabilized Q-learning with a neural network:
- Experience replay (Lin 1992): store experiences \((S_t,A_t,R_{t+1},S_{t+1})\) and learn from random samples of them. Each experience is used many times, and the correlation between consecutive experiences is broken.
- Target network: compute the update target \(R_{t+1}+\gamma\max_a\tilde q(S_{t+1},a)\) with a separate network that is copied from the main one every \(C\) updates. This reduces the instability of a target that moves with every step.
- Rewards were clipped to +1, −1, and 0, and the error term was clipped to \([-1,1]\). \(\varepsilon\) was decreased linearly over the first million frames.
Policy gradients
Instead of learning values and then choosing actions, you can represent the policy itself with parameters \(\boldsymbol\theta\) and move \(\boldsymbol\theta\) directly in the direction that improves performance. For discrete actions, a preference is computed for each action and turned into probabilities with a softmax (eq. 13.2). Such a policy can approach a deterministic one and handles continuous actions naturally (chapter 6). REINFORCE (Williams 1992) raises the probability of an action in proportion to the return that actually followed. Subtracting a baseline \(b(S_t)\) leaves the expected update unchanged while greatly reducing its variance. Use the state value as the baseline and the one-step TD error instead of the return, and you have an actor–critic method: the policy (actor) acts, and the value function (critic) evaluates the action.
\[\begin{aligned}\boldsymbol\theta&\leftarrow\boldsymbol\theta+\alpha\,\big(G_t-b(S_t)\big)\,\nabla\ln\pi(A_t\mid S_t,\boldsymbol\theta)\\ \boldsymbol\theta&\leftarrow\boldsymbol\theta+\alpha\,\delta_t\,\nabla\ln\pi(A_t\mid S_t,\boldsymbol\theta),\quad \delta_t=R_{t+1}+\gamma\hat v(S_{t+1},\mathbf{w})-\hat v(S_t,\mathbf{w})\end{aligned}\]REINFORCE with baseline (eq. 13.11) and one-step actor–critic (eq. 13.13). \(\nabla\ln\pi\) is the direction that most quickly raises the probability of that action.
PPO: not too far in one go
If a policy gradient step is too large, the policy can break in one update, and a broken policy collects bad data in a vicious circle. The PPO of Schulman et al. (2017) clips the objective so that there is nothing to gain from pushing the probability ratio \(r_t(\boldsymbol\theta)=\pi_{\boldsymbol\theta}(a_t\mid s_t)/\pi_{\boldsymbol\theta_{\text{old}}}(a_t\mid s_t)\) between the new policy and the old policy that collected the data outside \([1-\epsilon,1+\epsilon]\). That makes it safe to run several epochs of minibatch updates on the same data. The paper recommends \(\epsilon=0.2\) and computes the advantage \(\hat A_t\) with generalized advantage estimation (GAE), a sum of TD errors discounted by \(\gamma\lambda\). Simple and robust, it is widely used to train robot locomotion in simulation.
\[L^{\text{CLIP}}(\boldsymbol\theta)=\hat{\mathbb{E}}_t\Big[\min\big(r_t(\boldsymbol\theta)\hat A_t,\ \operatorname{clip}(r_t(\boldsymbol\theta),1-\epsilon,1+\epsilon)\,\hat A_t\big)\Big]\]PPO’s clipped objective (Schulman et al. eq. 7). The advantage \(\hat A_t\) says how much better than average the action was.
Seeing PPO’s clipping
One term of the objective, for a single time step, as a function of the probability ratio \(r\) (the paper’s figure 1). The dashed line is the unclipped \(r\hat A\). In the red region the gradient is zero, so there is no reason to push the policy further that way.
Balancing a pole
Keeping a pole hinged to a cart from falling over is an old testbed of reinforcement learning (Barto, Sutton & Anderson 1983; book example 3.4). Every 0.02 seconds the agent pushes the cart left or right. Tilting the pole past 12 degrees or moving the cart beyond 2.4 m is a failure; every step survived earns +1, so the return equals the time balanced. The experiment below uses the same physical constants and 500-step limit as OpenAI Gym’s CartPole-v1.
Training CartPole with PPO in your browser
The policy and the value function are small networks with two hidden layers of 32 neurons. After every 1,024 steps of data, PPO makes 8 passes of updates. The picture shows the current policy balancing the pole, and the chart shows how episode length changes. Usually it holds on for all 500 steps after about 20,000 steps.
Chapter 6
From Simulation to Reality
Textbook and papers: Sutton & Barto §13.7, 17.4 · Makoviychuk et al. (2021) · Liu et al. (2023) · Tobin et al. (2017) · Peng et al. (2018) · Amodei et al. (2016)
Continuous actions
The torques and velocities of a robot’s joints are continuous. Instead of a probability for each action, policy gradient methods learn the statistics of a probability distribution. The most common choice is a normal-distribution policy whose mean \(\mu(s,\boldsymbol\theta)\) and standard deviation \(\sigma(s,\boldsymbol\theta)\) come from a neural network (§13.7). Early in learning \(\sigma\) is large and the policy explores widely; as learning proceeds, \(\sigma\) shrinks and the policy becomes precise.
\[\pi(a\mid s,\boldsymbol\theta)=\frac{1}{\sigma(s,\boldsymbol\theta)\sqrt{2\pi}}\exp\!\left(-\frac{\big(a-\mu(s,\boldsymbol\theta)\big)^2}{2\,\sigma(s,\boldsymbol\theta)^2}\right)\]The normal-distribution policy for continuous actions (eq. 13.19).
Why simulation?
Real robots are slow, break down, and can get hurt while exploring. Collecting the 38 days of experience that DQN used in chapter 5 on a real robot would be hard. So much of robot learning happens in physics simulators.
- MuJoCo (Todorov et al. 2012): a physics engine that computes contact-rich robot dynamics quickly; it became a standard testbed for reinforcement learning research. The continuous-control experiments of the PPO paper also used MuJoCo.
- Isaac Gym / Isaac Sim (NVIDIA): run both the physics simulation and the network training on the GPU. Makoviychuk et al. (2021) reported training 2–3 orders of magnitude faster than the usual setup that shuttles between a CPU simulator and a GPU network.
- LIBERO (Liu et al. 2023): a benchmark of 130 language-specified robot manipulation tasks in four suites (Spatial, Object, Goal, and LIBERO-100). Every task comes with human teleoperated demonstrations, and chapter 11 uses it to compare VLAs.
The reality gap and domain randomization
A simulator cannot perfectly mimic friction, mass, latency, or what a camera sees. The difference that makes a policy that did well in simulation fail in reality is the reality gap. Domain randomization changes the simulator’s settings at random during training so that the policy works across all of those variations. Tobin et al. (2017) trained an object detector only on synthetic images with randomized textures, lighting, and camera positions, and it located real objects to 1.5 cm and supported grasping. With enough variability in the simulator, they argue, reality may look to the model like just another variation. Peng et al. (2018) randomized dynamics such as mass, friction, damping, and latency, and transferred a pushing policy to a real robot arm without any further training.
Randomize the mass and the policy holds up in reality
A robot must stop a load at a line 1 m ahead, with a wall 8 cm behind the line. The policy is a PD controller with two gains, tuned inside the simulator. One policy is tuned only to the simulator’s 1 kg mass; the other is tuned to many simulators with randomized masses at once. Both act with a 40 ms delay. Change the real mass and see what happens.
Reward hacking
Reinforcement learning agents find unexpected ways to get reward (§17.4). When the goal is far away and reward is sparse, it is tempting to add rewards for intermediate steps, but the book warns that such well-intentioned rewards can make the agent behave differently from what was intended and fail to achieve the real goal. Amodei et al. (2016) give the example of a cleaning robot. Reward it when it sees no messes, and it might disable its vision or cover messes with something it cannot see through. The book recommends leaving the reward alone and putting prior knowledge elsewhere, for example in the initial value function.
Circling created by an intermediate reward
The designer decided to give a bonus every time the robot enters the checkpoint C on the way to the goal G. Each step costs −0.1 and the goal gives +10. See where the optimal policy found by value iteration actually goes.
Chapter 7
Behavior Cloning and DAgger
Textbook and papers: Sutton & Barto §17.4 · Ross & Bagnell (2010) · Ross, Gordon & Bagnell (2011) · Zhao et al. (2023)
If you do not know what the reward should be but can watch someone who is good at the task, you can copy what they do. The book (§17.4) calls this imitation learning or learning from demonstration, and notes two routes: learn from the demonstrations directly with supervised learning, or recover a reward from them (inverse reinforcement learning). In robotics, demonstrations are often collected by a person teleoperating the robot.
Behavior cloning
Behavior cloning (BC) treats the (state, expert action) pairs from demonstrations as supervised training data and learns a function from states to actions. It is simple to implement and needs no reward. The problem is that it breaks the assumption supervised learning relies on, that training and test data come independently from the same distribution (Ross et al. 2011). The learner was trained on the states the expert passed through, but at test time it passes through states created by its own actions. One small mistake takes the robot into a state absent from the demonstrations, where a bigger mistake becomes likely. This is the compounding error problem.
Ross and Bagnell (2010) showed that a learner who errs with probability \(\varepsilon\) per step under the expert’s state distribution can make up to \(T^2\varepsilon\) mistakes over \(T\) steps under its own distribution. Mistakes can grow with the square of time rather than in proportion to it.
\[J(\hat\pi_{\text{BC}})\le J(\pi^*)+T^2\varepsilon\qquad\text{vs.}\qquad J(\hat\pi_{\text{DAgger}})\le J(\pi^*)+O(T\varepsilon)\]\(J\) is the cost (number of mistakes) over \(T\) steps. In the worst case, behavior cloning grows with the square of \(T\), DAgger in proportion to \(T\) (simplified from Ross et al. 2011, theorems 2.1 and 3.2).
DAgger: ask the expert where the learner went
DAgger (Dataset Aggregation) by Ross, Gordon, and Bagnell (2011) closes the distribution gap with data.
- Train a first policy on the expert demonstrations.
- Run the robot with the current policy (at first mixing in the expert’s actions with probability \(\beta_i\)).
- For every state the learner visited, ask the expert which action they would have taken there.
- Add all of the new data to the existing data (aggregate) and retrain the policy. Repeat steps 2–4.
The learner thus learns how to correct itself in the states it actually reaches. The paper demonstrated this on driving in Super Tux Kart and on playing Super Mario Bros. There is a cost: an expert has to stay on hand to label the learner’s states. For a person demonstrating by teleoperation, writing down actions for states the robot is already moving through is time-consuming and unnatural (Zhao et al. 2023). Variants that lighten the burden, such as having a person step in only when needed, have been studied as well.
Following a track: behavior cloning and DAgger
The expert is a controller that aims at a point ahead and drives twice around the middle of the track. The learner is a k-nearest-neighbor policy that looks at its distance from the centerline, its heading error, and the curvature ahead, and averages the actions of the 4 closest data points. Steering noise is added when it drives. Each press of “1 DAgger round” makes the learner drive a lap itself, and the expert’s action for every state it passed is added to the data.
Chapter 8
Actions with More Than One Right Answer
Papers: Zhao et al. (2023) ACT · Chi et al. (2023) Diffusion Policy · Ho et al. (2020) DDPM · Black et al. (2024) π0
The trap of the mean
Human demonstrations mix several right answers for the same situation. Sometimes the person goes around the obstacle in front of a cup on the left, sometimes on the right. Train a model to predict a single action with mean squared error, and it learns the average of the two answers: an action that drives straight into the middle of the obstacle. Both answers are good, but their average is bad. That is why the policy should be a generative model that represents the whole distribution of actions \(p(a\mid s)\). The Diffusion Policy paper names graceful handling of such multimodal distributions as the first advantage of diffusion policies.
Action chunks: ACT
Zhao et al. (2023) built ALOHA, a bimanual teleoperation system costing under $20k, and proposed ACT (Action Chunking with Transformers) on top of it. Instead of the next single action, the policy predicts a chunk of the next \(k\) actions, \(\pi_\theta(a_{t:t+k}\mid s_t)\). Deciding once every \(k\) steps cuts the effective horizon of the task by a factor of \(k\), which reduces the compounding error of chapter 7, and it also makes it easier to handle behavior the state alone cannot explain, such as a person pausing in the middle of a demonstration. Because human demonstrations differ a little every time, the distribution of chunks is learned with a conditional VAE. To keep the transition between chunks from jerking, ACT also uses temporal ensembling: it predicts a new chunk at every step and takes a weighted average of the overlapping predictions (weights \(w_i=\exp(-m\,i)\)). With this, it succeeded 80–90% of the time on 6 real tasks such as opening a translucent condiment cup and slotting a battery, from only 10 minutes, or 50, of demonstrations. The π0 paper, by contrast, reports that temporal ensembling hurt its performance, so it executed chunks as they were. The right choice depends on the task and the model.
Diffusion policies
The Diffusion Policy of Chi et al. (2023) applies the diffusion models used for image generation (Ho et al. 2020) to actions. In training, Gaussian noise of level \(k\) is mixed into a demonstrated action, and a network \(\varepsilon_\theta\) learns to predict that noise. To act, the policy starts from pure noise \(A^K\) and removes noise \(K\) times to produce an action chunk \(A^0\). Because the starting noise differs every time, the result settles cleanly into either the left mode or the right mode, even for the same observation.
\[\begin{aligned}\mathcal{L}&=\mathbb{E}\Big[\big\lVert\boldsymbol\varepsilon-\varepsilon_\theta\big(O_t,\ \sqrt{\bar\alpha_k}\,A^0_t+\sqrt{1-\bar\alpha_k}\,\boldsymbol\varepsilon,\ k\big)\big\rVert^2\Big]\\ A^{k-1}_t&=\alpha\big(A^k_t-\gamma\,\varepsilon_\theta(O_t,A^k_t,k)+\mathcal{N}(0,\sigma^2 I)\big)\end{aligned}\]Top: the noise-prediction objective (the simplified objective of Ho et al. 2020, conditioned on the observation \(O_t\)). Bottom: one denoising step (Chi et al. eqs. 1 and 4). Here \(\alpha,\gamma,\sigma\) are constants set by the noise schedule, not the discount rate of reinforcement learning.
The noise-prediction network effectively learns the gradient field of the action distribution, and denoising is like noisy gradient descent along that gradient (Chi et al. eq. 2). The policy looks at the last \(T_o\) steps of observations, predicts \(T_p\) steps of actions, executes only \(T_a\) of them, and predicts again. On real robots it produced actions in 0.1 seconds on an RTX 3080 with 100 training and 10 inference steps (DDIM), and across 15 tasks from 4 benchmarks it improved on existing methods by 46.9% on average.
Removing noise from a 2-D action distribution
The robot (bottom center) chooses an action \((\Delta x,\Delta y)\) that takes it around the obstacle (circle) to its next waypoint. The faint dots are demonstrated actions, and the ✕ is the mean action that mean-squared-error regression predicts. The 300 dark dots are samples of a diffusion policy that started from pure noise. Move the step slider or press Play to watch the noise clear.
The noise predictor in this experiment is not a trained network. It is the optimal predictor, which can be computed exactly when the demonstrations follow a Gaussian mixture. A real diffusion policy approximates it with a neural network.
Chapter 9
Robot Data
Papers: Ramos et al. (2021) RLDS · Open X-Embodiment Collaboration (2024) · Brohan et al. (2022) RT-1 · Kim et al. (2024) OpenVLA
Observations and actions
One step of robot learning data usually consists of an observation and an action. The observation holds images from fixed and wrist cameras, the robot’s own state (proprioception) such as joint angles and gripper state, and a language instruction such as “put the cup in the bowl.” What counts as the action differs from robot to robot.
- End-effector deltas: a 7-D action that combines the change in fingertip position \((\Delta x,\Delta y,\Delta z)\), the change in orientation (roll, pitch, yaw), and the gripper opening is common. RT-1’s arm action is exactly these 7 values, plus 3 for the mobile base and one for switching modes.
- Joint space: joint positions or velocities serve directly as the action. Bimanual robots such as ALOHA use joint positions, and the Franka experiments of the FPL paper used joint velocities.
Control rates differ too. RT-1 moved at 3 Hz, and the π0 paper notes that most public datasets use 2–10 Hz, while ALOHA teleoperates and records at 50 Hz. Mixing datasets therefore means aligning units, coordinate frames, and rates.
Normalization
Each dimension of an action has its own units and range: meters, radians, and gripper values between 0 and 1 share one vector. So each dimension is normalized separately. Dividing by the mean and standard deviation (z-score) and sending the minimum and maximum to −1 and 1 (min–max normalization) are common. Teleoperation data easily contains values that spike for an instant, and with min–max normalization one such outlier crowds all the other data into a narrow interval. That is why OpenVLA used the 1st and 99th percentiles as bounds. Whatever the method, the statistics used for normalization must be saved so that actions can be converted back to the original units when the robot executes them.
One outlier steals the resolution
This is one action dimension of teleoperation data (fingertip motion per step, in cm). Most values are small movements, with a very rare value that spikes for an instant. Look at the distribution mapped to \([-1,1]\) by three normalizations, and compare how many bins 98% of the data actually uses when the range is split into 256 bins as in chapter 10.
RLDS: a format for episodes
RLDS (Ramos et al. 2021) is a format for storing and sharing reinforcement learning and robot data without loss. A dataset is a collection of episodes, and an episode is a sequence of steps. Because it is built on TensorFlow Datasets, the same code reads many datasets. One step contains these fields:
| Field | Meaning |
|---|---|
observation | The observation at this step, held as a dictionary of images, robot state, language instruction, and so on |
action | The action taken after seeing this observation |
reward, discount | The reward and discount received after that action |
is_first | Whether this is the first step of the episode |
is_last | Whether this is the last step of the episode. Only the observation is defined at the last step. |
is_terminal | Whether the environment really ended here. An episode cut off by a time limit has is_last true but is_terminal false. |
The distinction in the last two fields mattered for CartPole in chapter 5 as well. Being cut off at the 500-step limit is not a failure, so training continued from the value of the next state.
Open X-Embodiment
Language models gained generalization from internet-scale text; can robots gain it by pooling data from many robots? Open X-Embodiment (2024) is a collection in which 21 institutions pooled 60 existing datasets from 34 research labs and converted them to the RLDS format. It covers 22 kinds of robots, from single arms to bimanual robots and quadrupeds, more than a million real robot trajectories, and 527 skills (160,266 tasks). RT-1-X, trained on this data, achieved a 50% higher success rate on average than each lab’s original method or RT-1 across several labs with little data. Experience from other robots helps. This collection becomes the training data of OpenVLA in chapter 10.
Chapter 10
How VLAs Are Built
Papers: Brohan et al. (2022) RT-1 · Brohan et al. (2023) RT-2 · Kim et al. (2024) OpenVLA
A vision–language model (VLM) is a large Transformer that takes images and text and outputs text. Pre-trained on internet-scale images and text, it knows the names of objects, spatial relations, and common sense. Transformers and attention are covered in chapter 14 of the Machine Learning and Deep Learning guide. RT-2 (Brohan et al. 2023) fine-tuned such a model to output robot actions instead of text, and called models of this kind vision–language–action (VLA) models.
Actions as text: tokenization
A language model outputs tokens from its vocabulary. To turn continuous actions into tokens, RT-1 and RT-2 split each action dimension into 256 uniform bins within that dimension’s range. An action then becomes a few integers between 0 and 255. RT-2 wrote the terminate command, three position changes, three rotation changes, and the gripper opening as a string of 8 integers such as “1 128 91 241 5 101 127.” In PaLI-X, which has a token for each number, it used those number tokens directly; in PaLM-E, which does not, it overwrote the 256 least frequently used tokens with action tokens. The paper reports that training on the original web data together with the robot data (co-fine-tuning), rather than on robot data alone, was important for generalization. The 55-billion-parameter model ran at 1–3 Hz on TPUs in the cloud.
\[\begin{aligned}b&=\min\!\Big(255,\ \Big\lfloor\frac{a-\ell}{h-\ell}\cdot 256\Big\rfloor\Big),\quad a\ \text{clipped to}\ [\ell,h]\\ \hat a&=\ell+\big(b+\tfrac12\big)\Delta,\qquad \Delta=\frac{h-\ell}{256},\qquad |a-\hat a|\le\frac{\Delta}{2}\end{aligned}\]Turning a one-dimensional action \(a\) into a bin index \(b\), and back into the bin’s center \(\hat a\). The bounds \([\ell,h]\) are each variable’s range in RT-1 and RT-2, and the 1st and 99th percentiles of the training data in OpenVLA.
OpenVLA
The OpenVLA of Kim et al. (2024) is a 7-billion-parameter VLA with open weights.
- Architecture: built on the pre-trained Prismatic-7B VLM. Images are read by concatenating the features of two vision encoders, SigLIP and DINOv2, which pass through a projector into the Llama 2 7B language model.
- Data: trained on 970k robot demonstrations selected from Open X-Embodiment.
- Action tokens: 256 bins per dimension, with bounds at the 1st and 99th percentiles so that outliers do not widen the bins. The Llama tokenizer reserves only 100 slots for new tokens, so the 256 least used tokens, the last 256 of the vocabulary, were overwritten with action tokens. Training is ordinary next-token prediction with a cross-entropy loss on the action tokens only.
- Performance: across 29 tasks and several robots, its absolute success rate was 16.5 percentage points higher than that of the 55-billion-parameter RT-2-X, with one seventh the parameters.
- Speed and fine-tuning: it runs at about 6 Hz in bfloat16 on a single RTX 4090. With LoRA it can be adapted to a new task by changing only 1.4% of its parameters, which takes 10–15 hours on a single A100.
A VLA that outputs actions as tokens generates them one after another, like a language model. A 7-D action needs 7 decoding steps. Chapter 11 is about making this fast.
256-bin quantization and its error
Here is a 100-step, 7-D action trajectory for picking up and moving an object. The selected dimension is split into \(N\) bins and decoded back to the bin centers, so you can compare the original values (line) with the decoded ones (steps). Add a spike and switch the bound method. The bottom row shows the 7 bin indices of the action at time \(t\).
Chapter 11
Fast and Precise VLAs
Papers: Kim, Finn & Liang (2025) OpenVLA-OFT · Black et al. (2024) π0 · Zhao et al. (2023) · Chi et al. (2023)
A VLA that outputs tokens one at a time, like OpenVLA, runs at 3–5 Hz. Delicate work with two arms needs control at 25–50 Hz or more (Kim et al. 2025). Here are two ways to close that gap.
OpenVLA-OFT: parallel decoding, action chunks, continuous actions
Kim, Finn, and Liang (2025) compared three design choices for adapting OpenVLA to a new robot—the decoding scheme (autoregressive vs. parallel), the action representation (discrete vs. continuous), and the learning objective (next-token prediction vs. L1 regression vs. diffusion)—and distilled them into the OFT (Optimized Fine-Tuning) recipe.
- Parallel decoding: feed the model empty action embeddings and replace the causal attention mask, which lets each token see only earlier ones, with bidirectional attention. All dimensions of the action then come out of a single forward pass. \(D\) sequential passes become 1.
- Action chunks: the same single pass produces \(K\) steps of actions (\(K=8\) on LIBERO). Latency rises by only 17% while throughput grows \(K\)-fold. It is the same idea as ACT in chapter 8.
- Continuous actions with L1 regression: instead of 256-bin tokens, a small MLP head outputs continuous values directly and is trained with the absolute error (L1) to the target. This worked better than discrete tokens, and as well as a diffusion objective while being much faster.
With this recipe the average success rate over LIBERO’s four suites rose from 76.5% to 97.1%, and action-generation throughput grew 26-fold. OFT+, which adds FiLM for better language following, ran at 25 Hz on the real bimanual ALOHA robot and beat π0, RDT-1B, Diffusion Policy, and ACT by up to 15 percentage points in average success rate.
| OpenVLA variant (A100, 7-D actions) | Throughput | Latency | LIBERO-Long success |
|---|---|---|---|
| Autoregressive (original OpenVLA) | 4.2 Hz | 0.240 s | 53.7% |
| + parallel decoding | 15.9 Hz | 0.063 s | – |
| + parallel decoding, chunk \(K=8\) | 108.8 Hz | 0.074 s | 86.5% |
| + continuous actions (L1 regression) | 109.7 Hz | 0.073 s | 90.7% |
| + continuous actions (diffusion, 50 inference steps) | 4.2 Hz | 1.907 s | 91.1% |
| + continuous actions (diffusion, 5 inference steps) | 35.1 Hz | 0.228 s | 90.0% |
Selected from Kim et al. (2025), table II. Throughput is the number of actions produced per second; latency is the time to produce one action (or chunk). The diffusion variant was trained with 50 steps, and cutting inference down to a single step brought its success rate to 0%.
π0: 50 Hz chunks with flow matching
The π0 of Black et al. (2024) is a 3.3-billion-parameter model that attaches a 300-million-parameter action expert to the 3-billion-parameter PaliGemma VLM. It was pre-trained on more than 10,000 hours of the team’s own robot data plus Open X-Embodiment, and it handled long tasks such as folding laundry. It produces actions not as tokens but through flow matching, a branch of diffusion: it learns the velocity that points from \(A_t^\tau\), a straight-line mix of the noise \(\boldsymbol\varepsilon\) and the target chunk \(A_t\), toward the target.
\[\begin{aligned}A_t^\tau&=\tau A_t+(1-\tau)\,\boldsymbol\varepsilon,\qquad \mathcal{L}(\theta)=\mathbb{E}\,\big\lVert v_\theta(A_t^\tau,o_t)-(A_t-\boldsymbol\varepsilon)\big\rVert^2\\ A_t^{\tau+\delta}&=A_t^\tau+\delta\,v_\theta(A_t^\tau,o_t)\end{aligned}\]The flow matching objective and inference. Integration starts from noise at \(\tau=0\) and runs to \(\tau=1\). π0 uses 10 steps (\(\delta=0.1\)) to produce an \(H=50\)-step chunk (Black et al., section IV).
The secret of its speed is splitting the computation. The images and the instruction go through the large VLM only once; with that result cached, only the small action expert runs 10 times. On an RTX 4090 with three cameras, the image encoders took 14 ms, the observation forward pass 32 ms, and the 10 flow matching steps 27 ms, 73 ms in total. On 50 Hz robots it executed 25 of the 50 steps in a chunk and then inferred again (every 0.5 seconds).
A latency calculator for decoding schemes
A simple model fitted to the measurements in the two papers above. The autoregressive, parallel, and iterative schemes are fitted to OpenVLA-OFT table II, measured on an A100; the π0 scheme is fitted to π0 table I, measured on an RTX 4090. Change the action dimension, the chunk size, and the number of iterative steps. Values outside the measured conditions are extrapolations.
Throughput is the number of actions that can be produced per second if the whole chunk is executed (\(K\)/latency); latency is the time from receiving a new observation to the first action. A larger chunk raises throughput, but while a chunk is executing the robot cannot react to new observations.
Chapter 12
From Preferences to Rewards
Papers: Bradley & Terry (1952) · Christiano et al. (2017) · Ouyang et al. (2022)
Writing a reward function for a robot that does a backflip, or that sets a cup down “carefully,” is hard. But show a person two behaviors and ask which is better, and they answer easily. Christiano et al. (2017) showed how to learn a reward function from a person’s comparisons of two short trajectory segments and then do reinforcement learning with that reward. They solved Atari games and simulated robot locomotion without a reward function, with human feedback on less than 1% of the agent’s interactions, and taught novel behaviors with about an hour of human time.
The Bradley–Terry model
Given a reward model \(\hat r\), the probability that a person prefers segment \(\sigma^1\) over \(\sigma^2\) is written in terms of the difference between the two segments’ summed rewards. This is the 1952 Bradley–Terry (BT) model. Christiano et al. liken it to Elo ratings in chess: just as the difference in two players’ ratings sets the probability that one beats the other, the difference in two segments’ predicted rewards sets the probability that a person picks one. The reward model is trained to reduce the cross-entropy between these probabilities and the person’s actual choices.
\[\begin{aligned}\hat P\big[\sigma^1\succ\sigma^2\big]&=\frac{\exp\sum_t \hat r(o^1_t,a^1_t)}{\exp\sum_t \hat r(o^1_t,a^1_t)+\exp\sum_t \hat r(o^2_t,a^2_t)}=\sigma\big(\hat R(\sigma^1)-\hat R(\sigma^2)\big)\\ \mathcal{L}(\hat r)&=-\sum\Big[\mu_1\log\hat P\big[\sigma^1\succ\sigma^2\big]+\mu_2\log\hat P\big[\sigma^2\succ\sigma^1\big]\Big]\end{aligned}\]Christiano et al. (2017) eq. 1 and its loss (§2.2.3). \(\mu\) is the label that puts 1 on the segment the person chose, and \(\sigma(\cdot)\) is the sigmoid.
RLHF
This approach went on to wide use in large language models as reinforcement learning from human feedback (RLHF). InstructGPT (Ouyang et al. 2022) goes through three stages. It is first fine-tuned on model answers written by people (SFT), then a reward model is trained from rankings of several answers to the same prompt, and finally the policy is trained with PPO against that reward model. A KL penalty keeps the policy from moving too far from the SFT model. As a result, answers from the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3. Robotics applies the same idea to behaviors. A learned reward model is not perfect, though, so the policy can exploit its gaps and the reward hacking of chapter 6 can happen again. The KL penalty is also a guard against that.
Learning a reward model from pairwise comparisons
Here are two paths a robot arm might take to carry a cup past a vase. Choose the one you like better. The reward model is a weighted sum of three features (speed, smoothness, distance from the vase), retrained with the Bradley–Terry loss after every choice. Instead of choosing yourself, you can hand the job to a virtual rater with hidden criteria.
Chapter 13
DPO and Multi-Axis Preferences
Papers: Rafailov et al. (2023) DPO · Torne, Mahajan, Bhat & Finn (2026) FPL
Reward maximization held back by KL
The reinforcement learning stage of RLHF is the problem of raising reward without straying too far from a reference policy \(\pi_{\text{ref}}\) (usually the SFT model). \(\beta\) sets the length of that leash. Rafailov et al. (2023) start from the fact that the optimal policy of this problem has a closed form.
\[\begin{aligned}&\max_{\pi}\ \mathbb{E}_{y\sim\pi(\cdot\mid x)}\big[r(x,y)\big]-\beta\,\mathbb{D}_{\text{KL}}\big[\pi(y\mid x)\,\Vert\,\pi_{\text{ref}}(y\mid x)\big]\\ &\pi_r(y\mid x)=\frac{1}{Z(x)}\,\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{1}{\beta}r(x,y)\Big)\end{aligned}\]KL-constrained reward maximization (DPO eq. 3) and its optimal solution (eq. 4). \(Z(x)\) is the partition function that makes the probabilities sum to 1.
Solving eq. 4 backward writes the reward in terms of the policy: \(r(x,y)=\beta\log\frac{\pi_r(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x)\). The Bradley–Terry model uses only the difference between two answers’ rewards, so the hard-to-compute \(Z(x)\) cancels. The result is the DPO (Direct Preference Optimization) loss, which trains the policy directly on preference data instead of training a separate reward model and running reinforcement learning. The policy itself plays the part of an implicit reward model.
\[\mathcal{L}_{\text{DPO}}(\pi_\theta;\pi_{\text{ref}})=-\mathbb{E}_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)\Big]\]The DPO loss (eq. 7). \(y_w\) is the preferred answer and \(y_l\) the dispreferred one.
Steering a policy with β
A robot has 8 ways to grasp a cup. The gray bars are the reference policy \(\pi_{\text{ref}}\) learned from demonstrations, and the blue bars are the optimal policy of eq. 4. The reward model hides one error: it rates option 7, “hooking only the tip of the handle,” far better than it really is. Try lowering \(\beta\).
The limit of a single overall preference
So far, a preference has been a single judgment: which one is better overall. But if one trajectory is faster and the other safer, it is hard to pick one overall. The longer the task, the larger this ambiguity grows, and different quality criteria get blurred into one signal.
FPL: freeform, multi-axis preferences
FPL (Freeform Preference Learning) by Torne, Mahajan, Bhat, and Finn (2026) lets annotators define the axes of comparison themselves in natural language, such as “speed,” “safety,” “quality of placement,” or “carefulness,” and pick the better of two full trajectories along each axis.
- Reward model: instead of a separate model per axis, a single reward model is trained that takes the axis description \(l_k\) as a condition. The point is to let one model handle similar wordings such as “speed,” “fast,” and “efficient.” A pre-trained VLM scores each prefix of the trajectory, and the sum of those scores is the trajectory’s reward. The loss is a Bradley–Terry loss applied to each axis.
- Policy: rather than collapsing the axes into one reward with a weighted sum, it trains a reward-conditioned policy that imitates actions given the axis descriptions and the trajectory’s per-axis rewards as conditions. The per-axis rewards are standardized over the whole dataset, and at execution time the desired scores steer the policy. The paper fine-tuned the π0.5 VLA this way.
- Iteration: collecting preferences → training the reward model → training the policy → running the new policy to add data can be repeated.
\[\begin{aligned}\mathcal{L}_{\text{reward}}(\phi)&=-\mathbb{E}\Big[\sum_{k}\log\sigma\Big((2y_k-1)\big(r_\phi(\tau_i\mid l_k)-r_\phi(\tau_j\mid l_k)\big)\Big)\Big]\\ \mathcal{L}_{\text{policy}}(\theta)&=-\mathbb{E}_{\tau_i}\Big[\sum_t\log\pi_\theta\big(a_t\mid s_t,\ l_1,r_\phi(\tau_i\mid l_1),\dots,l_{K},r_\phi(\tau_i\mid l_{K})\big)\Big]\end{aligned}\]FPL’s per-axis reward learning (Torne et al. eq. 3) and reward-conditioned policy learning (eq. 5). \(y_k\) is 1 if \(\tau_i\) was preferred on axis \(l_k\).
On 4 real robot tasks (putting a cube into a target bowl, folding shorts, plating toast, setting up a table) and 2 simulated tasks, FPL’s success rate was on average 38 percentage points higher than the second-best baseline. The paper also reports two properties. One is compositionality. In a task where a nut must be placed on the right peg quickly, the data held fast and slow demonstrations for the left peg but only slow ones for the right peg, yet the FPL policy achieved “right peg, fast,” which was not in the data. The other is steerability: without retraining, changing only the conditioning scores could send the same policy to the left peg.
Steering a policy with conditioning scores
A toy version of the two-peg task from the FPL paper. Each dot represents one demonstrated trajectory by two properties of its behavior: which peg it uses and how fast it is. The data contains only left–fast, left–slow, and right–slow. The blue policy takes the standardized scores of the two axes as separate conditions. The orange policy takes only one score, a person’s judgment of which trajectory is better overall. Because the target task is placing on the right peg, we assume this score is set mostly by whether the peg is right, with speed counting only a little (peg 1, speed 0.25). Both policies fit a linear model from conditioning scores to behavior.
The real FPL uses a VLA policy and a VLM reward model, and runs the two-peg task with a robot arm in a simulator. This experiment only shows, in its simplest form, the idea that splitting the axes into separate conditions can reach combinations that are not in the data.
Further reading
- The rest of Sutton & Barto: Monte Carlo methods (chapter 5), n-step bootstrapping (chapter 7), combining planning and learning (chapter 8), and function approximation (chapters 9–11)
- The basics of neural networks, Transformers, and diffusion models: the Machine Learning and Deep Learning guide
- Using brain signals to control robots: the Brain–Computer Interfaces guide
Appendix A
Glossary
No matching terms. Try an abbreviation or a different word.
Appendix B
Textbook and Paper Map
This table matches the chapters of this guide to the sections of Sutton & Barto’s 2nd edition and to the papers each chapter rests on. From chapter 7 on, the material is recent work that the textbook does not cover, so it relies mainly on the original papers.
| Chapter | Sutton & Barto (2018) | Original papers |
|---|---|---|
| 1 What is robot learning? | 1.1, 1.3, 3.1, 17.3–17.4 | – |
| 2 MDPs and rewards | 3.1–3.5 | – |
| 3 Value functions and the Bellman equation | 3.5–3.6, 4.1–4.4, 4.6 | – |
| 4 Q-learning and exploration | 2.2, 6.1, 6.4–6.5 | Watkins 1989 |
| 5 Deep reinforcement learning | 13.1–13.5, 16.5 | Mnih et al. 2015, Schulman et al. 2016, 2017 |
| 6 From simulation to reality | 13.7, 17.4 | Todorov et al. 2012, Makoviychuk et al. 2021, Liu et al. 2023, Tobin et al. 2017, Peng et al. 2018, Amodei et al. 2016 |
| 7 Behavior cloning and DAgger | 17.4 | Ross & Bagnell 2010, Ross et al. 2011 |
| 8 Actions with more than one right answer | – | Zhao et al. 2023, Chi et al. 2023, Ho et al. 2020 |
| 9 Robot data | – | Ramos et al. 2021, Open X-Embodiment 2024, Brohan et al. 2022 |
| 10 How VLAs are built | – | Brohan et al. 2022, 2023, Kim et al. 2024 |
| 11 Fast and precise VLAs | – | Kim et al. 2025, Black et al. 2024 |
| 12 From preferences to rewards | – | Bradley & Terry 1952, Christiano et al. 2017, Ouyang et al. 2022 |
| 13 DPO and multi-axis preferences | – | Rafailov et al. 2023, Torne et al. 2026 |
Textbook
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. free official edition from the authors
Papers
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv:1606.06565
- Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13(5), 834–846.
- Black, K., et al. (2024). π0: A vision-language-action flow model for general robot control. arXiv:2410.24164
- Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345.
- Brohan, A., et al. (2022). RT-1: Robotics Transformer for real-world control at scale. arXiv:2212.06817
- Brohan, A., et al. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control. CoRL. arXiv:2307.15818
- Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2023). Diffusion Policy: Visuomotor policy learning via action diffusion. RSS. arXiv:2303.04137
- Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. NeurIPS. arXiv:1706.03741
- Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS. arXiv:2006.11239
- Kim, M. J., et al. (2024). OpenVLA: An open-source vision-language-action model. CoRL. arXiv:2406.09246
- Kim, M. J., Finn, C., & Liang, P. (2025). Fine-tuning vision-language-action models: Optimizing speed and success. arXiv:2502.19645
- Liu, B., et al. (2023). LIBERO: Benchmarking knowledge transfer for lifelong robot learning. NeurIPS Datasets and Benchmarks. arXiv:2306.03310
- Makoviychuk, V., et al. (2021). Isaac Gym: High performance GPU-based physics simulation for robot learning. arXiv:2108.10470
- Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. (early version: arXiv:1312.5602)
- Open X-Embodiment Collaboration (2024). Open X-Embodiment: Robotic learning datasets and RT-X models. ICRA. arXiv:2310.08864
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS. arXiv:2203.02155
- Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. ICRA. arXiv:1710.06537
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. NeurIPS. arXiv:2305.18290
- Ramos, S., et al. (2021). RLDS: An ecosystem to generate, share and use datasets in reinforcement learning. arXiv:2111.02767
- Ross, S., & Bagnell, J. A. (2010). Efficient reductions for imitation learning. AISTATS.
- Ross, S., Gordon, G., & Bagnell, J. A. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS. arXiv:1011.0686
- Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. ICLR. arXiv:1506.02438
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347
- Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. IROS. arXiv:1703.06907
- Todorov, E., Erez, T., & Tassa, Y. (2012). MuJoCo: A physics engine for model-based control. IROS.
- Torne, M., Mahajan, A., Bhat, A., & Finn, C. (2026). Freeform preference learning for robotic manipulation. arXiv:2606.32027
- Watkins, C. J. C. H. (1989). Learning from delayed rewards (PhD thesis). University of Cambridge.
- Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256.
- Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023). Learning fine-grained bimanual manipulation with low-cost hardware. RSS. arXiv:2304.13705