35 · Reward design and reward hacking


Part	8 · Reinforcement learning for robots
Run type	CPU · Mac or Colab CPU
Prerequisites	33
Learn first	MDPs, States, Actions & Rewards
Time	~60 min
Code	`experiments/35-reward-design/experiment.py`
Status	✅ done

Act on this lesson

Tick each step when you have done it. Saved in this browser only.

01 · Read The problem and the core idea; answer the questions before you read.
02 · Build it Type the steps yourself, then run uv run python experiments/35-reward-design/build.py
03 · Run the experiment Predict first, then run uv run python experiments/35-reward-design/experiment.py
04 · Break it Change the one parameter and watch it fail.
05 · Check yourself Quizzes and exercises at the end.

The agent optimizes the reward you wrote, not the task you meant. Every term, and every way an episode can end, is part of what you wrote.

The problem

The Go2 training config sums ten reward terms with nonzero weights, from tracking_lin_vel = 1.0 to action_rate = -0.01 and torques = -0.0002, and it carries one line that exists only because of how an agent exploits rewards: only_positive_rewards = True # if true negative total rewards are clipped at zero (avoids early termination problems) (legged_robot_config.py, go2_config.py). On the SO-101 reach, with a fixed PPO and budget, this lesson compares reward shapes, produces a reward hack on purpose (a policy that scores well on the written reward without doing the task), and sweeps a smoothness penalty.

Before you read · 2 questions. Guess first; the lesson tells you why.

1. The reward is the positive closeness $1 - \tanh(d/5\,\text{cm})$ and the episode ends the moment $d < 2$ cm. What did the trained arm learn?

To reach every target as fast as possible.
To park about 3 cm from the target and keep collecting reward, because entering the ball ends all future payments.
To oscillate around the target so that the shaping term pays twice.

Why: Part B: 95/300 success, and run without termination the same policy ends at a median 30.8 mm away. Its written return (22.67) beats the scripted IK that reaches every target (2.28).

2. Every step of an episode costs a penalty (a negative reward), and falling over ends the episode. What might the agent learn?

To fall early, because ending the episode stops the penalties.
To stay up as long as possible, because falling is always bad.
Nothing different; termination does not affect the return.

Why: After a terminal state every reward is 0, so with $r < 0$ per step ending is a gain. The Go2 config clips negative totals at zero for exactly this, “avoids early termination problems”.

Core idea

In plain words

A reward is a metric, and RL optimizes it with no common sense. Pay per line of code and you get long code. Pay an arm for being close to the target and end the game the moment it arrives, and it learns to hover just outside, because arriving stops the payments.

You have four kinds of building blocks. A sparse reward pays only on success: exact, but silent early on, like a test suite whose one end-to-end test never passes yet. A dense reward pays a little every step for being closer: informative, but a proxy. Shaping adds hints; the safe kind (potential-based) pays for progress and takes it back on retreat, so going back and forth earns nothing the original reward would not pay. Regularizers are small penalties for jerky or expensive motion; they trade success for smoothness.

How an episode ends is part of the reward: after the end every future reward is zero. If the per-step reward is positive, ending is a loss and the agent avoids it; if it is negative, ending is a gain and the agent seeks it.

Where termination flips the incentive

flowchart TD
    S["state after a step"] --> T{"does the episode<br/>end here?"}
    T -- "no" --> R["collect reward:<br/>task + shaping − penalties"]
    R --> S
    T -- "yes" --> Z["every future reward is 0"]
    Z --> Q{"sign of the<br/>per-step reward"}
    Q -- "positive" --> H["ending is a loss:<br/>hover near the goal"]
    Q -- "negative" --> E["ending is a gain:<br/>rush in, or fall early"]

Shaping and termination, as code

# Potential-based shaping: F = gamma * Phi(s') - Phi(s), with Phi = -20 * distance
def shaping(dist, dist_prev, gamma=0.95):
    return gamma * (-20 * dist) - (-20 * dist_prev)   # > 0 when closer; also 20*(1-gamma)*dist when standing still

# It cannot be farmed: summed along any trajectory it telescopes to the two endpoints,
#   sum_t gamma^t * F_t = gamma^T * Phi(s_T) - Phi(s_0)

# What ending the episode costs: the discounted value of staying forever
def value_of_staying(r_per_step, gamma=0.95):
    return r_per_step / (1 - gamma)   # ending forfeits this: a relief if r < 0, a loss if r > 0

Term	Example (reach, distance $d$ in m)	Role
sparse task	$\mathbb 1[d < 0.02]$	says exactly what success is; gives no gradient until success is found
dense task	$-d$	points toward the goal at every step; defines “good” by a proxy
potential-based shaping	$F = \gamma\Phi(s') - \Phi(s)$	adds guidance without changing the optimal policy
regularizers	$-w\lVert a_t - a_{t-1}\rVert^2$ , $-w\lVert\tau\rVert^2$	trade task performance for smoothness and energy

$d$ is the gripper-to-target distance, $\mathbb 1[\cdot]$ is 1 when the condition holds and 0 otherwise, $\Phi$ is any function of the state (the potential), $s$ and $s'$ are the states before and after a step, $\gamma$ the discount, $w$ a penalty weight, $a_t$ the action and $\tau$ the joint torque.

Math details: why potential-based shaping is safe and why termination matters

Potential-based shaping. Ng, Harada and Russell showed that adding $F(s,s') = \gamma\Phi(s') - \Phi(s)$ for any potential $\Phi$ leaves the optimal policy unchanged, and that, without further knowledge of the MDP, this form is also necessary for that guarantee (ICML 1999, pp. 278–287). Along a trajectory the terms telescope, $\sum_t \gamma^t F_t = \gamma^T \Phi(s_T) - \Phi(s_0)$ , a bonus that depends only on the endpoints, so shaping changes how fast you learn, not what is optimal. We use $\Phi = -20\,d$ with the PPO discount $\gamma = 0.95$ .

Termination. A policy that could stay in a state forever while collecting $r$ per step would earn $\sum_{t\ge0}\gamma^t r = r/(1-\gamma)$ . Entering a terminal state replaces that with 0. For $r > 0$ that is a loss the agent learns to avoid; for $r < 0$ it is a gain the agent learns to seek. In locomotion, falling terminates the episode, so a total made mostly of negative penalties makes an early fall attractive; the Go2 config clips the total at zero for “early termination problems”. The reverse case is our hack.

Go deeper: MDPs, States, Actions & Rewards; Ng, Harada, Russell, ICML 1999 (link above).

Build it

You will now write the reward terms, check that shaping telescopes on an SO-101 episode, solve a small corridor exactly, and put numbers on the termination hack. Nothing is trained: Steps 2, 4 and 5 use the reach task with the scripted IK controller, Step 3 a tabular toy; about 2 s.

The functions are in physai/rl.py; uv run python experiments/35-reward-design/build.py runs the five steps (output in build_output.txt).

Step 1 · The reward terms

def sparse_reward(d, radius=0.02):
    """1 inside the success ball, 0 elsewhere."""
    return float(d < radius)

def closeness_reward(d, scale=0.05):
    """Positive at every step, 1 at the target: the reward behind the termination hack."""
    return 1.0 - np.tanh(d / scale)

def potential_shaping(phi, phi_prev, gamma):
    """Potential-based shaping term F = gamma * Phi(s') - Phi(s) (Ng, Harada, Russell 1999)."""
    return gamma * phi - phi_prev

def action_rate_penalty(a, a_prev, w):
    """Smoothness regularizer: -w * ||a_t - a_(t-1)||^2."""
    return -w * float(np.sum((np.asarray(a) - np.asarray(a_prev)) ** 2))

Composed as in the Experiment (e.g. sparse plus potential_shaping(-20 * d, -20 * d_prev, 0.95)) and compared with its reward functions on 1000 random steps:

== Step 1. Reward terms vs the Experiment's reward functions, 1000 random steps ==
sparse 0.0e+00, pbrs 4.4e-16, closeness 0.0e+00, action rate 0.0e+00
closeness at d = 0, 2, 5, 30.8 mm: 1.000, 0.620, 0.238, 0.452

The closeness reward is 0.620 at the 2 cm radius and 0.452 at 30.8 mm, where the hacking policy parked.

Step 2 · Shaping telescopes

def discounted_sum(rewards, gamma):
    """sum_t gamma^t r_t."""
    return float(np.sum(np.asarray(rewards) * gamma ** np.arange(len(rewards))))

Collect only the shaping term along one episode, once with the scripted IK and once with random actions:

== Step 2. Shaping telescopes: sum_t gamma^t F_t = gamma^T Phi(s_T) - Phi(s_0), Phi = -20 d ==
   scripted IK: d  55.2 ->   0.2 mm, sum gamma^t F_t = +1.104336, gamma^T Phi_T - Phi_0 = +1.104336, diff 2.2e-16
random actions: d  55.2 -> 147.0 mm, sum gamma^t F_t = +1.055855, gamma^T Phi_T - Phi_0 = +1.055855, diff 4.4e-16
loop 10 cm -> 5 cm -> 10 cm, gamma 1.00: shaping earned +0.0000 (= (gamma^2 - 1) Phi = +0.0000)
loop 10 cm -> 5 cm -> 10 cm, gamma 0.95: shaping earned +0.1950 (= (gamma^2 - 1) Phi = +0.1950)

Both sums equal $\gamma^T\Phi(s_T) - \Phi(s_0)$ to rounding: the shaping a trajectory earns depends only on its endpoints. A loop 10 cm → 5 cm → 10 cm earns 0 with $\gamma = 1$ but +0.1950 with $\gamma = 0.95$ , the formula’s $(\gamma^2 - 1)\Phi$ . Not a loophole: every path between the same endpoints earns the same discounted bonus, so the loop is worth what the original reward says. Step 3 checks this in the optimal policy.

Step 3 · Which rewards keep the optimal policy

A corridor of 11 cells 1 cm apart, the target in cell 10 (entering it ends the episode), actions left, stay, right, $\gamma = 0.95$ . Value iteration solves it exactly:

def value_iteration(next_state, reward, terminal, gamma, iters=2000):
    """Optimal values and greedy actions of a deterministic tabular MDP.

    next_state[s, a] and reward[s, a] describe one step; entering a terminal state ends the episode (value 0 after).
    """
    V = np.zeros(next_state.shape[0])
    for _ in range(iters):
        Q = reward + gamma * np.where(terminal[next_state], 0.0, V[next_state])
        V = Q.max(1)
    return V, Q.argmax(1)

== Step 3. A corridor of 11 cells, 1 cm apart; cell 10 is the target and ends the episode ==
reward                         greedy action in cells 0..9   value of cell 0
sparse: 1 on arrival             >>>>>>>>>>             0.630
sparse + gamma Phi' - Phi        >>>>>>>>>>             2.630
sparse + 0.5 per step closer     >>>>>>>>><             6.768
closeness 1 - tanh(d/5cm)        >>>>>>>>>o            12.251
(> right, < left, o stay)
shaped - sparse values, cells 0..9: max |difference - (-Phi(s))| = 3.3e-16

Sparse and shaped rewards give the same policy, right in every cell, and the shaped values are the sparse ones minus $\Phi(s)$ , as Ng, Harada and Russell predict (difference 3.3e-16). Two equally reasonable-looking rewards change the policy. Paying 0.5 per step closer, without taking it back, makes cell 9 step left: going back and forth earns more than arriving. The closeness reward with termination makes cell 9 stay, one centimetre short, forever: Experiment B’s hack, solved exactly.

Step 4 · The termination hack in numbers

== Step 4. The termination hack in numbers ==
scripted IK, closeness reward, ends on success: written return 2.28, mean length 10.1 steps
parking at 30.8 mm for 80 steps: 0.452 per step, written return 36.13; discounted value of parking forever r / (1 - gamma) = 9.03
entering the ball pays at most 1.00 once, then nothing

The scripted controller reaches every target in 10.1 steps and is paid 2.28, the Experiment’s number. Parking at 30.8 mm pays 0.452 per step, 36.13 over 80 steps, 9.03 in discounted value, against at most 1.00 once for arriving. The trained policy’s 22.67 lies between: it parks in most episodes, not all.

Step 5 · How large is a smoothness penalty?

Measure both terms of $r = -d - w\lVert a_t - a_{t-1}\rVert^2$ on the scripted controller’s episodes:

== Step 5. Action-rate penalty vs the task term, scripted IK, 100 episodes ==
per step: mean d = 11.9 mm, mean ||a_t - a_t-1||^2 = 0.0082
w = 0.03: penalty / task term = 0.02
w = 0.1 : penalty / task term = 0.07
w = 0.3 : penalty / task term = 0.21
w = 1.0 : penalty / task term = 0.68
w = 3.0 : penalty / task term = 2.05
Go2 config: action_rate = -0.01 times dt 0.02 s -> 0.0002 per step

For these actions the penalty is 2% of the task term at $w = 0.03$ , 21% at 0.3, 68% at 1.0; Break it’s success collapses between 0.1 and 0.3. The IK’s actions are not the trained policy’s, so this gives the scale only. The Go2’s action_rate = -0.01 becomes 0.0002 per step (times the 0.02 s period).

What you keep

physai.rl now also has sparse_reward, closeness_reward, potential_shaping, action_rate_penalty, discounted_sum and value_iteration. Lesson 36 builds a locomotion reward from terms of this kind. tests/test_35_reward_design.py checks the telescoping and, on 20 random MDPs, that potential-based shaping keeps the optimal policy and shifts the values by $-\Phi$ ; run uv run pytest tests/test_35_reward_design.py.

Use it

Gymnasium leaves the reward to the environment; to change it without editing the env, wrap it. gymnasium.wrappers.TransformReward(env, func) maps each scalar reward (scaling, offsets). Shaping needs the state, so it goes in a wrapper’s step (or ReachEnv’s reward_fn), with the learner’s $\gamma$ . Unitree’s training code builds the reward as a weighted sum of named terms, each scale multiplied by the policy period (legged_robot.py):

Piece	This lesson / physai	`legged_gym`
a term	a function of `info`	a method `_reward_<name>`, weighted by `scales.<name>`
weight per step	$w$ as written	`self.reward_scales[key] *= self.dt`
action smoothness	`action_rate_penalty`: $-w\lVert a_t - a_{t-1}\rVert^2$	`_reward_action_rate`: `torch.sum(torch.square(self.last_actions - self.actions), dim=1)`
positive clip	not used	`only_positive_rewards`: `torch.clip(self.rew_buf[:], min=0.)`
termination term	not used	`_reward_termination`: `self.reset_buf * ~self.time_out_buf`, added after the clip

The last row links lessons 32 and 35: the termination term fires on real terminations, not time-outs, and is added after the clip, so a termination penalty survives it.

Three traps:

Termination flips the incentive. With a positive per-step reward ending is a loss (Step 3: one cell short), with a negative one a gain; decide which you want before adding a terminal condition.
Shaping with the wrong $\gamma$ is not potential-based. $F = \gamma'\Phi(s') - \Phi(s)$ with $\gamma' \ne \gamma$ no longer telescopes in the learner’s discounted return, and the guarantee is gone.
Weights are relative. A penalty weight means nothing without the task term’s size (Step 5); after a change of control rate, legged_gym’s per-step weights change too, because they are multiplied by dt.

Check after building · 2 questions.

1. In Build it, Step 3, the corridor pays 0.5 for every step closer to the target and nothing back for a step away. What does the optimal policy do next to the target?

It enters the target, as with the sparse reward.
It stays in cell 9 forever.
It steps left in cell 9, because going back and forth earns more than arriving.

Why: That bonus is not potential-based, so it changes the optimal policy (value of cell 0 6.768 against 0.630 for the sparse reward). Potential-based shaping keeps the policy and shifts the values by $-\Phi(s)$ (difference 3.3e-16).

2. In Step 2, the loop 10 cm → 5 cm → 10 cm earns +0.1950 of shaping with $\gamma = 0.95$ . Can the arm farm the shaping term by going back and forth?

Yes; +0.1950 per loop adds up without limit.
No; every path between the same endpoints earns the same discounted bonus, $\gamma^T\Phi(s_T) - \Phi(s_0)$ , so the loop is worth what the original reward says.
Only with $\gamma = 1$ .

Why: The +0.1950 is the formula’s $(\gamma^2 - 1)\Phi$ ; on the real episodes both sums match $\gamma^T\Phi(s_T) - \Phi(s_0)$ to rounding, and in the corridor the shaped policy equals the sparse one.

Experiment

experiment.py keeps everything from lesson 33 fixed (absolute joint targets, 50 Hz, PPO reference config, 300k steps, 3 seeds × 100 evaluation episodes) and swaps the reward function passed to the env.

The rewards are built from Build it’s Step 1 terms: sparse, sparse + shaping, closeness, and $-d$ + action-rate penalty.

Predict: with the closeness reward and an episode that ends the moment the gripper is within 2 cm, what will the trained arm do?

MuJoCo 3.17.0, NumPy 2.5.4, PyTorch 2.14.1 (CPU), Apple Silicon Mac (Apple M5). Deterministic per seed. The whole script (33 runs, six in parallel) took 1904 s wall-clock on a heavily loaded machine. python experiment.py --only close+end,pbrs trains just those cells for all three seeds; this rerun reproduced their lines exactly. --quick trains seed 0 only, which reproduces the first “success per seed” entry of every row but not the pooled columns.

A. Reward shape (no termination; success = within 2 cm at the 1.6 s time limit):

reward	success per seed	pooled	Wilson 95%	median final distance
dense $-d$	100 / 98 / 100	298/300	97.6–99.8%	7.5 mm
sparse $\mathbb 1[d < 2\,\text{cm}]$	8 / 0 / 0	8/300	1.4–5.2%	117.8 mm
sparse + $0.95\,\Phi(s') - \Phi(s)$ , $\Phi = -20d$	99 / 97 / 99	295/300	96.2–99.3%	7.8 mm

(Median final distance: mean over the three seeds.) The sparse reward is the honest statement of the task and it fails: the initial policy rarely gets within 2 cm (lesson 32: 1 of 200 random-action episodes), so for most of training every reward is zero and the advantages carry no information. A potential-based term restores learning to the dense reward’s level (overlapping intervals) without changing which policy is optimal (Build it, Step 3).

B. Reward hacking. The episode now ends (terminated) the moment $d < 2$ cm, which looks like a natural way to define “reached”. The reward is the positive closeness term $1 - \tanh(d / 5\,\text{cm})$ :

reward, termination	success per seed	pooled	Wilson 95%	written return	episode length	median final distance
closeness, ends on success	23 / 43 / 29	95/300	26.7–37.1%	22.67	62.3 steps	28.8 mm
$-d$ , ends on success	99 / 100 / 98	297/300	97.1–99.7%	−0.72	9.2 steps	14.7 mm
closeness, no termination	98 / 100 / 100	298/300	97.6–99.8%	64.00	80.0 steps	5.1 mm

Two control measurements make the hack explicit:

The scripted IK controller, which reaches every target, scores a written return of only 2.28 under the closeness-plus-termination reward (Build it, Step 4). The hacking policy scores 22.67: it beats the correct controller on its objective by doing the task less.
Run without termination, the same hacking policy ends at a median of 30.8 mm from the target: it parks the gripper about 3 cm away, just outside the success ball, and collects roughly 0.45 per step for the rest of the episode.

The mechanism is the diagram’s: entering the ball ends all future positive reward, so the agent approaches and stops short (the corridor of Build it, Step 3 does the same). The 95 successes are episodes in which it still ended inside the ball. The same termination rule with a negative reward ( $-d$ ) gives the opposite incentive: the agent hurries in (9.2 steps on average) because every step costs. Positive reward without termination also works (298/300). The hack lives in the combination, which is why it is easy to write by accident.

Break it

Smoothness penalties are where the trade-off becomes a cliff. Sweep the weight $w$ of the action-rate term (the Go2 config uses the same term, action_rate = -0.01, computed as $\sum (a_{t-1} - a_t)^2$ in legged_robot.py; _prepare_reward_function there multiplies every scale by the policy period, 0.02 s, so its per-step weight is $0.01 \times 0.02 = 0.0002$ against a velocity-tracking term of at most $1.0 \times 0.02$ per step):

$w$	success per seed	pooled	Wilson 95%	median final distance (mm)	torque rate (N·m/s)	power (W)
0	100 / 98 / 100	298/300	97.6–99.8%	7.5	1002	9.66
0.03	81 / 99 / 99	279/300	89.5–95.4%	8.8	933	9.44
0.1	80 / 87 / 96	263/300	83.5–90.9%	11.4	901	9.45
0.3	41 / 9 / 94	144/300	42.4–53.6%	25.6	850	8.20
1.0	7 / 35 / 14	56/300	14.7–23.5%	39.2	674	6.54
3.0	27 / 6 / 26	59/300	15.6–24.5%	31.9	743	5.64

Left: success against the action-rate penalty weight w, from 298/300 at w 0 through 263/300 at 0.1 down to about a fifth at w 1.0 and 3.0, with per-seed points spread from 9 to 94 at w 0.3. Right: torque rate falls only from 1002 to between 674 and 850 N·m/s over the same range.

Up to $w = 0.1$ the policy loses 6 to 12 percentage points of success (279/300 and 263/300 against 298/300) and gains little smoothness (torque rate 1002 → 901 N·m/s). From $w = 0.3$ on, success collapses and the seeds disagree wildly (9 vs 94 out of 100): the penalty is now a sizable share of the distance term (Build it, Step 5: about a fifth for the scripted controller’s actions at $w = 0.3$ ). Power falls (9.66 → 5.64 W) largely because the arm moves less: the median final distance grows to 25.6–39.2 mm. Two lessons: set a regularizer’s weight relative to the task term, and with a stiff servo, smoothing the policy’s targets barely smooths the torque. If smooth torque is the goal, change the interface or the gains (lesson 34) before raising $w$ .

On the real robot

Penalty weights trade task success for gentler motion in simulation; whether the measured torque-rate reduction carries over depends on the real actuator. On the SO-101 it was not measured.
A termination rule written for training is also a safety rule on hardware: on a real robot “episode ends” means a supervisor stops the motion (lesson 15). The reward must not make that stop something the policy seeks or avoids for its own sake.
The Go2 config’s weights were tuned for its simulator and PD gains; change either and they mean something else.

Check yourself

Quick check · 2 questions. Click an answer.

1. The sparse reward gave 8/300; adding $0.95\,\Phi(s') - \Phi(s)$ with $\Phi = -20d$ gave 295/300. Why does this term not change what the optimal policy is?

It pays the same bonus at every step regardless of the state.
It replaces the success condition with a distance condition.
Summed along any trajectory it telescopes to $\gamma^T\Phi(s_T) - \Phi(s_0)$ , which depends only on where the trajectory starts and ends.

Why: Ng, Harada and Russell (Math details). Shaping changes how fast you learn, not what is optimal; part A shows the speed-up.

2. An action-rate penalty of weight $w = 0.3$ gave 144/300 (seeds 41 / 9 / 94) while the torque rate only fell from 1002 to 850 N·m/s. What is the takeaway?

Set the weight relative to the task term’s scale, and with a stiff servo change the interface or gains if smooth torque is the goal.
Penalties always smooth the torque in proportion to their weight, so raise $w$ further.
The seeds disagree, so the penalty has no effect at all.

Why: Break it table: up to $w = 0.1$ success falls to 263/300 with little smoothing (901 N·m/s); from 0.3 on it collapses. Smoothing the targets of a stiff servo barely smooths its torque.

Easy. For the closeness reward, what per-step reward does a gripper parked 3 cm from the target collect, and what does it collect after entering the success ball?
Medium. Fix the hack without removing the termination: add a terminal bonus $B$ when the episode ends by success. Estimate the $B$ that makes reaching at least as good as hovering for the remaining ~60 steps, then train three seeds and measure.
Hard. Replace the action-rate penalty by a torque-rate penalty computed inside the env at the physics rate, and repeat the sweep. Does it buy more smoothness per lost success than the action-rate term?

Answers

$1 - \tanh(0.03/0.05) = 1 - \tanh(0.6) \approx 0.46$ per step; after success the episode is over, so every later reward is 0.
Hovering for $n$ more steps at ~0.46 is worth $0.46 \sum_{t<n} 0.95^t$ , approaching $0.46/(1-0.95) \approx 9.2$ (Build it, Step 4: 9.03 at 30.8 mm); a bonus well above that removes the incentive. We did not train it.
Open; we did not run it.

Takeaways and sources

Sparse rewards state the task exactly but gave 8/300 within the budget; potential-based shaping recovered 295/300 without changing the optimal policy.
Termination plus positive per-step reward is a measured reward hack: 95/300 success, the gripper parked ~3 cm away, and a written return (22.67) about ten times that of a controller that reaches every target (2.28). Check the sign of the reward wherever an episode can end.
Regularizer weights fall off a cliff: action-rate weight 0.1 cost 12 percentage points (263/300), 0.3 halved success; set weights relative to the task term and measure the smoothness you actually get.

Next: lesson 36, legged locomotion and massively parallel training.

Sources

A. Y. Ng, D. Harada, S. Russell, “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping”, ICML 1999, pp. 278–287. mlanthology
unitree_rl_gym · legged_robot_config.py, go2_config.py, legged_robot.py (_reward_action_rate, _reward_torques) (BSD-3-Clause).
J. Schulman et al., “Proximal Policy Optimization Algorithms”, 2017. arXiv 1707.06347
Gymnasium · Env API (terminated vs truncated).
Code: experiment.py, output.txt; env and PPO from lesson 32 and lesson 33; figure from plot.py.