▶ Watch the promo video on YouTube (1:28)

Sim2Real policy transfer with a DIY wheeled goose robot. This was a student project for DD2600 at KTH Royal Institute of Technology in 2026.

We built a two-wheeled self-balancing robot from about 850 SEK of parts. We trained a neural-network controller for it entirely in simulation and ran that network on the robot's ESP32-S3 microcontroller. The robot has no hand-tuned PID loop. The learned policy reads the IMU and wheel encoders 100 times a second and writes the motor PWM directly. It balances the robot and follows forward-speed and turn-rate commands from a locally run browser interface.

The same policy in simulation (left) and on the real robot (right). MP4 version.

Reinforcement learning can find controllers that are hard to tune by hand. A policy that works in simulation, though, often fails on the real robot. The model is never exact, the sensors are noisy and late, and cheap parts vary from one unit to the next. This is the Sim2Real gap.

We set out to cross that gap deliberately, under tight constraints:

• Cheap hardware. The robot uses hobby-grade N20 gear motors with backlash, a 6-axis IMU, and a 3D-printed chassis.
• A tiny computer. The whole controller has to run on a microcontroller, so it gets no GPU and no Linux, only a 240 MHz dual-core chip.
• Proprioception only. The robot senses nothing of its environment. It knows only its own tilt, rotation rates and wheel speeds.

The goal from the project proposal was a policy that follows linear and angular velocity commands in the robot's own frame. It should work on flat ground first, then handle disturbances and harder surfaces. To get there, we relied on domain randomization and curriculum learning in simulation.

The real robot and its CAD model.

Circuit design for the goose elecctronics in KiCad.

The chassis and wheels are 3D printed. A strip cut from a bike inner tube, pulled over each wheel, gives it grip. In the Fusion 360 assembly, every electrical component is modelled roughly where it sits in the robot. Each part gets a material density that reproduces its measured weight. The assembly is then exported to MuJoCo with the ACDC4Robot add-in. This keeps mass and centre of mass close to the real robot, which matters a lot for a balancing robot.

→ Details, print tips and export steps are in hardware-design/.

The robot is simulated in full 3D with MuJoCo MJX. 4,096 copies run in parallel on one GPU and are trained with Brax PPO in JAX. A full training run takes about 25 minutes on a laptop.

The critic can "see" the exact friction, pushes and mass, so it learns an accurate value function quickly. The actor never sees those quantities. It has to infer them from how the robot has reacted over the last 80 ms, and the critic's value estimates guide it there. This approach replaced the teacher-student (DAgger) pipeline from the proposal: it trains the small network directly and avoids the distribution shift of imitation learning (ARCHITECTURE.md).

The reward has an alive bonus and a penalty on pitch rate (wobble) but not on pitch angle, because leaning is how the robot accelerates. Velocity and yaw-rate tracking use Gaussian kernels. Small penalties on motor effort and action jitter protect the real motors. The reward never changes during training; only the task gets harder. This avoids catastrophic forgetting between curriculum stages (REWARDS.md).

All 4,096 environments advance together. Finished episodes are pooled in windows of 1,000 and checked against the stage's test: survival, a speed ratio (are we actually going as fast as commanded?), drift when told to stand still, and survival through reversals.

Training over 200M steps. Dotted lines mark curriculum stage advances. "Last stage" is evaluated with full randomization the whole time, so it shows how robustness grows even before training reaches that stage.

A 10 s evaluation episode under randomization: commanded vs. actual forward speed and yaw rate, pitch, and the two motor commands.

→ Setup, training, evaluation, keyboard teleop and export are covered in software_stack/.

The firmware (ESP32_software/robot) is written with ESP-IDF in plain C.

• Core 1 runs only the control loop, triggered by a hardware timer at 100 Hz. Each tick it reads the IMU and encoders, builds the 11-value observation frame, shifts it into the 8-frame history, runs the forward pass and writes the PWM.
• Core 0 handles everything else: the WiFi access point, the HTTP server and WebSocket, the web page, and log streaming. The control task never prints to the UART itself, because one log line at 115200 baud would stall the loop.

A self-checking policy file. make export writes a single policy.bin. It contains the weights and the observation layout, plus the physical constants the policy was trained with (control rate, command limits, half-track, motor no-load speed, tilt limit). It also carries CRCs and 8 simulated test vectors. Before the robot uses a policy, it checks four things:

1. The file is intact.
2. The observation layout and constants match the firmware.
3. The firmware's own forward pass reproduces the stored actions.
4. Only then can the robot be started.

A mismatched sign, unit or field order in a single input is enough to make the robot fall, so we wanted those mistakes caught at boot rather than on the floor.

The web interface. Join the robot's WiFi (goose) and open 192.168.4.1. From there you can:

• drive with an on-screen pad or the arrow keys;
• calibrate the IMU;
• watch live telemetry (observation frame, loop jitter, inference time);
• download every run as a 100 Hz CSV;
• upload a new policy over WiFi with no reflashing; it persists in a dedicated flash partition;
• replay a recorded or scripted command sequence.

The browser interface served by the robot: joystick, live telemetry and the device log.

Safety. The motors cut out when:

• tilt passes 0.6 rad, the limit the policy was trained up to;
• the IMU stops responding;
• the browser stops sending commands.

The data_analysis/ scripts take a CSV downloaded from the robot and replay the same command sequence in the simulator. They plot both on the same axes, so every deviation between the real robot and its model is visible directly.

uv run data_analysis/plot_logs.py   LOG.csv   # plot a robot log
uv run data_analysis/compare_log.py LOG.csv   # overlay robot (solid) and simulation (dashed)
uv run data_analysis/evaluate_runs.py LOG.csv # tracking and smoothness metrics per run

The clip at the top of this page (MP4) shows the trained policy balancing on a desk next to the same run in simulation.

Balancing (zero velocity command) and forward driving (linear velocity command).

Spinning in place (angular velocity command) and driving up a ramp (robot only).

Forward run on the real robot: default policy (left) vs. policy trained without randomization (right), which oscillates throughout.

• Redesign the 3D model for buttons and LEDs. Add mounts for physical buttons and status LEDs to the chassis, so the robot can be started, stopped and calibrated, and can show its state, without opening the browser interface.
• Design a PCB. Turn the KiCad schematic into a custom PCB that replaces the current wiring and holds the ESP32-S3, motor driver and buck converter.

The project is split into separate repositories in this organization.

# Train a policy (NVIDIA GPU)
cd software_stack
make install-cuda && make train      # ~25 min on a laptop GPU
make teleop                          # drive it yourself in simulation
make export                          # → models/policy.bin

# Flash the firmware
source ~/.espressif/tools/activate_idf_v6.1.sh
cd ../ESP32_software/robot
idf.py -p /dev/ttyACM0 build flash monitor

# Join WiFi "goose" (password "goosegoose"), open http://192.168.4.1,
# calibrate, check the observation signs, and press Start.
# New policies can be uploaded from the same page.

Philipp Seidel · Josef Kahoun · Laura Mª Quirós · Filippo Di Fiore

All four of us worked on every stage of the pipeline.