Physical AI Engineering from Scratch

EXP_000 · 44 LESSONS · 10 PARTS · 2026OPEN SOURCE · MIT + CC BY 4.0

PHYSICAL AIFROM SCRATCH

44 lessons from a physics engine to a robot that picks up what you ask for. Every lesson has a runnable experiment; every number comes from a real run.

Simulation, kinematics, control, perception, planning, imitation learning, RL, sim-to-real and robot foundation models. Runs on a laptop.

Start the course Reading paths Code on GitHub

LEARN IN YOUR TERMINAL

npx skills add fport/physical-ai-from-scratch
> Use start-learning to get a study plan.

CLAUDE CODE · CODEX · CURSOR · + ANY SKILL.MD AGENT

Your coding agent becomes your tutor: a short placement quiz, a plan, then one lesson at a time with the experiments run on your machine.

FIG_001 · INVERSE KINEMATICSbrowser sketch, not MuJoCo

A two-link arm solves IK every frame to reach, grasp and place a cube. Lessons 08–10, 16.

FIG_PD · INTERACTIVE SIMULATION

Inside a joint controller

A motor holds an arm against gravity with two numbers: stiffness Kp and damping Kd. Turn them down and the arm sags; remove damping and it rings; switch on gravity feedforward and the sag disappears.

BREAK IT IN THE LESSON →

Browser toy model, not MuJoCo. Measured results are in the lesson.

Kp (stiffness) Kd (damping) gravity feedforward

sag overshoot damping ratio

FIG_DT · INTERACTIVE SIMULATION

Inside a simulation step

A physics engine jumps time forward in steps and only checks for collisions at those instants. Make the step larger and a falling cube can end up on the other side of a thin table.

BREAK IT IN THE LESSON →

Browser toy model, not MuJoCo. Measured results are in the lesson.

timestep (ms)

jump per step

FIG_FLT · INTERACTIVE SIMULATION

Inside a sensor filter

Averaging the last n readings calms a noisy sensor but reports where the joint was, not where it is. Find the window where noise and lag cost the same.

BREAK IT IN THE LESSON →

Browser toy model, not MuJoCo. Measured results are in the lesson.

window n (samples) motion speed

noise left lag total RMS error

A complete, open curriculum for the engineering behind robots that act in the physical world: simulation, spatial math, kinematics, dynamics and control, perception, planning, imitation learning, reinforcement learning, sim-to-real, and robot foundation models. Every lesson has a runnable experiment and every number comes from a run or a linked primary source.

Who it is for

Engineers and students who can program in Python and want to build robot software end to end. General math, machine learning, deep learning and reinforcement-learning basics are not re-taught here: each lesson links the matching lesson of the free AI Engineering from Scratch course under Learn first, and this book teaches the robotics-specific part on top of it.

How it is built

44 lessons in 10 parts and a capstone, in strict prerequisite order. Each lesson is 30–90 minutes.
One template: the problem → core idea → experiment → break it → on the real robot → check yourself → takeaways and sources.
Runnable on a laptop. Almost every experiment runs on a Mac or Colab CPU with MuJoCo; the few that need a GPU or the real arm say so in their metadata (run type CPU / GPU / ART / HW).
Recurring robots: the SO-101 arm, the Franka Panda and the Unitree Go2, using the MuJoCo Menagerie models.

Map

Part 1 · SimulationHow a physics engine describes a world and steps it forward, and which numbers decide whether the result is trustworthy. · Lessons 01–05

Part 2 · Spatial mathRepresenting orientation and pose without ambiguity, and moving points between frames. · Lessons 06–07

Part 3 · KinematicsFrom joint angles to the gripper pose and back, with velocities and the places where the math breaks. · Lessons 08–10

Part 4 · Dynamics and controlWhy bodies move the way they do, and how controllers turn a target into torques safely. · Lessons 11–16

Part 5 · Perception and estimationTurning noisy sensors and camera pixels into a state the controller can trust. · Lessons 17–21

Part 6 · Planning and optimal controlFinding collision-free motions and computing controls by optimization instead of hand-tuning. · Lessons 22–24

Part 7 · Learning from demonstrationTeaching a policy by showing it, and measuring honestly whether it learned. · Lessons 25–31

Part 8 · Reinforcement learning for robotsLearning behavior from reward in simulation, and the design choices that make it work on robots. · Lessons 32–36

Part 9 · Sim to realMeasuring and closing the gap between simulation and hardware. · Lessons 37–40

Part 10 · Foundation models and agentsLarge pretrained policies, language-model agents that call robot skills, and learned world models. · Lessons 41–43

CapstonePutting the whole stack together on one task. · Lessons 44

Learn with your coding agent

npx skills add fport/physical-ai-from-scratch

This installs four skills from skills/ into your coding agent (Claude Code, Codex, Cursor or any agent that reads SKILL.md): start-learning (a short placement quiz and a study plan in PHYSICAL-AI-LEARNING.md), learn (one lesson per session: intuition, diagram, code, a prediction, the experiment run on your machine, exercises), check-understanding (a quiz on a part or lesson) and course-guide (where is a topic taught, what to read first). In Claude Code start with /start-learning.

Run the experiments

git clone https://github.com/fport/physical-ai-from-scratch.git
cd physical-ai-from-scratch
uv sync                                   # Python 3.14 + locked dependencies
uv run python experiments/02-timestep/experiment.py
uv run python experiments/09-jacobians/build.py   # a lesson's Build it steps, checked against MuJoCo
uv run pytest                             # tests for physai/, the library you build in the lessons

To read the book as a website locally: cd site && npm install && npm run dev (site/README.md).

Honesty

Every number in the book comes either from an experiment in experiments/ (its exact output is saved next to it) or from a linked primary source. Nothing that was not run is presented as a result; hardware steps that have not been run on the real arm are labelled as such. Rules: Numbers and sources.

Videos

The Turkish short-video series Koddan Kasa summarizes lessons from this book; see Videos and lessons.

Acknowledgements

The lesson format and the “raw version before the framework” principle are inspired by Rohit Ghumare and contributors’ AI Engineering from Scratch (MIT), which this book also uses as its prerequisite course. No text or code was copied.

License

Code (experiments/, notebooks/, tools/, site/) is MIT (LICENSE); the lesson text is CC BY 4.0 (LICENSE-CONTENT). Robot models are downloaded from MuJoCo Menagerie under their own licenses.