Teaching the robot to see your hand

Post 01 gave the robot a voice; Post 02 gave it a reflex. Now the grand finale: sight. We'll have it watch through its camera, ask an AI — in plain English — what your hand is doing, and respond with a matching move and a sound.

Every AI app, from the fanciest assistant down to this, is really the same three steps: capture something, ask the AI about it, act on the answer. Once your kid sees that shape here, they'll spot it everywhere.

• 👀 A robot that reads your hand gesture (wave, thumbs-up, peace…) with an AI vision model.
• 🤖 It responds with a matching gesture and the giggle from the earlier posts.
• 🧩 The three ideas behind every AI app: capture → ask → act.

Build it one piece at a time — each prompt gets one thing working before the next. That's the real lesson: big scary projects are just small clear steps in a row.

The finished robot is the Get the app button up top: install it, add your OpenAI key on the settings page, and wave. Here's the sound it plays when it spots you (from Posts 01–02):

The heart of it, in two snippets. First, grab what the camera sees:

from reachy_mini import ReachyMini
from PIL import Image

with ReachyMini() as robot:
    frame = robot.media.get_frame()                       # camera image (numpy array, BGR)
    Image.fromarray(frame[:, :, ::-1]).save("view.jpg")   # flip to RGB, save it
    print("saved view.jpg -- that's what the robot sees")

Then ask the AI what's in the photo — this is the whole "brain":

import base64
from openai import OpenAI

client = OpenAI()                                    # reads your OPENAI_API_KEY
img = base64.b64encode(open("view.jpg", "rb").read()).decode()
reply = client.chat.completions.create(
    model="gpt-5.6-luna",
    max_completion_tokens=2000,                      # newer models use this, not max_tokens
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What hand gesture is the person making? One word."},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
    ]}],
)
print(reply.choices[0].message.content)              # e.g. "wave"

The full app stitches all four pieces together and hardens them: a background thread does the looking so the motion never stutters, your API key is kept safe (never shown to the model or the browser), and a settings page lets you pick the model and the detection sound. It's all open — read the complete code in the app repo.

Where next? You've built a robot that sees, thinks, and responds. That's… a robot. Go wave at it — and show someone.

Optional, and not about the robot — but of the three builds, this is the one that earns a proper conversation. To recognise your hand, the robot takes a photo in your home and sends it over the internet to a company's computer. That's not a scary thing; it's just a real thing, and a kid who builds it is old enough to know it.

Then talk about it:

• 📷 Our robot only looks when it sees movement, and only sends one frame. Is that enough? What would you change?
• 👻 Who should get to say yes before a camera points at someone — the person holding it, or the person in the picture?
• 🏠 Is there a room in your home where this robot shouldn't go? Who decides?