Robot Academy
Summer Curriculum
Six weeks. One robot that grows every week. You start with a rolling mecanum car controlled from code. You end with an autonomous system that recognizes faces, moves a robotic arm, and completes its own missions. Every gate requires you to explain your work — not just run it.
Contents
Week 1 — Foundation Build Week 2 — PID & Vision Week 3 — FPV Command Week 4 — Face Recognition Week 5 — Robotic Arm Week 6 — Full DemoFoundation Build
What You Build
A fully functional mecanum wheel robot car controlled from Python over SSH. The car moves in all four directions (including sideways strafing) using keyboard input. An HC-SR04 ultrasonic sensor detects obstacles and stops the car automatically before collision.
📚 Reading Assignments (before Day 3)
- Simon Monk — Raspberry Pi Cookbook: Chapters on GPIO, Motors, Servos, Sensors
- Simon Monk — I2C and SPI sections (for future weeks)
- Mike Cook & Timmons-Brown — Learn Robotics with Raspberry Pi: Pi setup, GPIO basics, PWM motor control, HC-SR04 wiring
Pi Setup & SSH
- Flash Raspberry Pi OS Lite, enable SSH via boot partition flag
- First SSH session: hostname, apt update/upgrade, configure static IP
- Install Python tools:
python3-venv, create project venv - GPIO basics: GPIO.setmode(BCM), digital output to LED, verify wiring
PWM & Motor Control
- Read: Cook's motor and PWM chapters
- Wire H-bridge to Pi GPIO (IN1/IN2/ENA for each motor)
- Write PWM class: set duty cycle, change direction, stop
- Test: spin each motor at 25%, 50%, 75%, 100%
Mecanum Drive Logic
- Study mecanum wheel kinematics: roller angles, force decomposition
- Implement 4-wheel drive mixer: forward/backward, strafe left/right, rotate
- Command line input: WASD + Q/E for rotation
- Verify sideways strafe — the distinctive mecanum move
Ultrasonic Sensor
- Wire HC-SR04: voltage divider on Echo pin (1kΩ + 2kΩ)
- Implement distance measurement with time.perf_counter()
- Add median filter (5 readings) to reduce noise
- Integrate with drive: auto-stop if distance < 15cm
Polish & Gate Review
- Code cleanup: constants, functions, proper main guard
- systemd service for auto-start on Pi boot
- Review gate questions with Mirk — don't guess, explain
- Reading check with Mirk (5 questions from the books)
🔒 Week 1 Gate — 5 Questions (via Mirk)
What is SSH, and why does a robotics engineer need to know how to use it? Not "it lets you connect to another computer" — explain why that specifically matters when you're building a robot.
What is PWM and what does "duty cycle" mean? Give a real example of how changing the duty cycle changes what the motor does.
How does a mecanum wheel robot move sideways? Which wheels spin which direction, and why does that produce sideways motion?
What's the difference between a servo and a regular DC motor? If you wanted to move a joint to exactly 45° and hold it there, which would you use and why?
What is a virtual environment in Python and why do we use one? What problem does it solve that just installing packages globally would cause?
PID Control & Computer Vision
What You Build
Two behaviors: (1) An orbit car that uses PID to maintain exactly 30cm from a wall while circling. You tune Kp, Ki, Kd by watching the oscillation. (2) A color-tracking turret arm — a 2-DOF servo arm that keeps a red ball centered in the camera frame using HSV color detection and centroid tracking.
📚 Reading Assignments
- Timmons-Brown: Pi Camera chapter (CSI port, picamera2 setup, capture_array)
- Cook: Sensor noise chapter (median filters, why sensors lie)
- Cook: Motor chapter — speed values vs PWM duty cycle
PID Theory & Implementation
- Study PID conceptually: P overshoot, D damping, I steady-state
- Implement PIDController class with configurable Kp/Ki/Kd/dt
- Test with simulated error values before connecting hardware
Orbit Behavior
- Ultrasonic sensor → PID → motor speed correction
- Tune P only first: find oscillation, back off, add D
- Add I term last — only if steady-state error remains
Pi Camera & OpenCV
- Set up picamera2, capture to numpy array
- BGR vs RGB — understand the channel order mismatch
- Convert to HSV, use cv2.inRange to create binary mask
- Find contours, filter by area, draw bounding circle
Tracking Arm
- Wire 2 servos (base + tilt) to Pi via PCA9685 PWM board
- Compute centroid from largest contour
- Map pixel X offset → base servo angle, Y offset → tilt angle
- Add dead zone: don't move arm if centroid is within ±20px of center
Integration & Gate
- Run both behaviors independently, then combined on same robot
- Tune HSV thresholds for your specific ball and lighting
- Gate review with Mirk — reading check included
🔒 Week 2 Gate — 5 Questions + Reading Check
What are the three terms in a PID controller and what does each one do? Don't just give the formula — explain in plain language what problem each term solves.
What happens if you set the proportional gain too high? What does the robot physically do, and why?
Why do we use HSV color space instead of RGB when trying to detect a colored object? Give a specific example of when RGB would fail.
What is a contour in OpenCV and how do you find one? Walk through the steps from raw camera frame to having a list of contours.
What is a centroid and how does your code use it? Describe what the centroid tells you and how you connected it to the arm's base servo.
FPV Command
What You Build
A fully remote-controlled FPV robot. You pair a PS4/Xbox controller to the Pi via Bluetooth. The camera streams live MJPEG video to your browser. You pilot the robot from a phone or laptop looking at the stream — your robot sees what you see.
Bluetooth Setup
- Pair gamepad using bluetoothctl (scan, pair, trust, connect)
- Verify with
ls /dev/input/andevtest - Map axes and buttons: print raw pygame joystick values
Controller Integration
- Apply dead zone (±0.1) to all axis readings
- Map left stick → mecanum forward/strafe, right stick → rotation
- Add turbo button: L2 trigger doubles speed multiplier
- Emergency stop: circle button sends all motors to 0
MJPEG Stream Server
- Flask server on port 5000 serving multipart HTTP stream
- Capture frame with picamera2, encode to JPEG, send as part
- Serve a simple HTML page with
<img src="/stream"> - Test stream in browser on same WiFi network
FPV Integration & Latency Testing
- Run controller script + stream server together (two terminals or threads)
- Measure stream latency: film a stopwatch, compare timestamps
- Tune JPEG quality (60–80%) for latency vs image quality tradeoff
- Add camera overlay: battery indicator, speed readout via cv2.putText
Field Test & Gate
- Obstacle course navigation: cones, door frames, narrow gaps
- Time yourself completing a course with and without video feed
- Gate review with Mirk
🔒 Week 3 Gate — 5 Questions
What is Bluetooth pairing vs. Bluetooth connecting? Why do you have to pair once but connect every time?
What is a dead zone in controller input and why do you need one? What goes wrong without it?
What does MJPEG stand for and how does the stream work? Describe in plain terms what the server is actually sending over the network.
What is latency and why does it matter for FPV driving? At what latency does it start to feel uncomfortable, and why?
Your stream server runs on port 5000. What is a port? Why does the server need one, and what would happen if two programs tried to use the same port?
Face Recognition & State Machines
What You Build
The "Herald" — an autonomous robot that searches a room for a specific person, drives to them, and delivers a message (plays a text-to-speech audio file). It uses the face_recognition library to identify who it's looking at, ArUco markers for reliable navigation waypoints, and a state machine to manage its behavior (IDLE → SEARCHING → FOUND → DELIVERING → RETURNING).
Face Detection vs Recognition
- Install face_recognition (dlib-based): takes 30+ min on Pi, start early
- Build face database: take 10+ photos of each known person, generate encodings
- Test recognition: print identified name + confidence in real time
ArUco Markers
- Print ArUco markers (4x4_50 dictionary), laminate 3 for durability
- Detect markers: cv2.aruco.detectMarkers + estimatePoseSingleMarkers
- Use marker distance for proximity detection (0.3m = "arrived")
State Machine Design
- Define states: IDLE, SEARCHING, FOUND, DELIVERING, RETURNING
- Implement as Python Enum + dict-based transition table
- Design search pattern: expanding spiral covering the room
Integration
- Combine face recognition + state machine + motor control
- Add text-to-speech: espeak or pyttsx3 for the delivery message
- Test: place target person in room, start robot, verify full cycle
Demo Run & Gate
- Film the complete herald cycle — SEARCHING to DELIVERED
- What breaks? Fix the most common failure mode
- Gate review with Mirk
🔒 Week 4 Gate — 5 Questions
What is the difference between face detection and face recognition? Which one is harder and why?
What is a face encoding in the face_recognition library? What does it actually represent, and why can you compare two encodings to see if they're the same person?
What is an ArUco marker and what advantage does it have over color-based detection? Give a real scenario where color detection would fail but ArUco wouldn't.
What is a state machine? Draw (or describe in words) the states and transitions in the herald behavior.
The herald behavior uses a search pattern instead of random movement. Why? What problem does a systematic pattern solve that random movement doesn't?
Robotic Arm & Inverse Kinematics
What You Build
A 3-DOF robotic arm mounted on the car (shoulder, elbow, wrist + gripper). The arm computes inverse kinematics to reach any point in its workspace. You calibrate it using an optimization loop that learns the actual link lengths from measured positions. Final demo: arm picks up a colored block identified by the camera.
Dynamixel Setup & Forward Kinematics
- Install DynamixelSDK, configure baud rate (1Mbps), scan servo IDs
- Write angle_to_position() and position_to_angle() converters
- Implement forward_kinematics(θ1, θ2): returns (x, y) tip position
- Verify: command known angles, measure physical tip position
Inverse Kinematics
- Implement law-of-cosines IK solver for 2-joint arm
- Add is_reachable() guard — fail gracefully outside workspace
- Use atan2 (not atan) — understand why for full-quadrant coverage
- Test: command target positions, verify arm reaches them
Calibration Optimizer
- Collect 10 training pairs: commanded position → measured tip position
- Implement cost function: sum of squared errors across all training points
- Use scipy.optimize.minimize (Nelder-Mead) to find calibrated L1, L2, offsets
- Compare arm accuracy before and after calibration
Camera-to-World Mapping
- Place calibration grid at known world coordinates, record pixel positions
- Fit pixel→cm transform using numpy least squares
- Integrate: camera detects colored block → compute world position → IK → move arm
Pick and Place Demo + Gate
- Full cycle: detect block, compute position, reach with arm, close gripper
- Film from overhead and side camera simultaneously
- Gate review with Mirk
🔒 Week 5 Gate — 5 Questions
What is forward kinematics? Give a concrete example using your arm's actual link lengths.
What is inverse kinematics and why is it harder than forward kinematics? What are the two main failure cases where IK can't find a solution?
What does atan2 do differently than atan? Why does this matter in robotics specifically?
Explain the camera-to-world coordinate mapping. Why can't you just use pixel coordinates directly as arm targets?
In the training loop, what does the optimizer actually do? Explain in plain terms what "minimize total error over the parameter space" means, and how the arm improves as a result.
Full Autonomous Demo
What You Build
Everything combined in one autonomous demo mission. The robot starts from a fixed position, navigates to find a specific person, delivers a message, then returns and uses its arm to pick up and deposit an object at a target location. The full mission runs without touching the robot or code.
Integration Sprint
- Combine Week 4 herald behavior + Week 5 arm into single state machine
- New states: NAVIGATE_TO_PICKUP, PICK_UP, NAVIGATE_TO_DROP, DEPOSIT
- Robust error handling: what happens if arm misses the block?
Rehearsal Runs
- 10 full mission runs — record success rate and failure modes
- Fix the two most common failure modes
- Add a "mission report" print at end: success/fail, time elapsed, states visited
Film the Demo
- Clean the robot (it'll be on camera)
- Film 3 full successful runs from fixed tripod angle
- Film one "behind the scenes" with SSH terminal visible
Written Reflection (the Week 6 Gate)
- Write a 3-5 paragraph reflection covering: what you built, what was hardest, what you'd build next, and one concept you now understand that you didn't at the start
- Submit to Mirk — he'll evaluate it
🏁 Week 6 Gate — Written Reflection
No quiz questions for Week 6. Instead, write a 3–5 paragraph reflection covering these four things:
1. What you built across all 6 weeks — the full arc from blinking an LED to autonomous arm control. Be specific: what components, what code.
2. What was the hardest concept to understand and how did it finally click? (PID? IK? State machines? Something else?)
3. What broke most, and what does that tell you about real engineering projects?
4. What would you build next if you had 6 more weeks?
Submit via Mirk. He'll ask follow-up questions on anything you didn't fully explain.