How AI Learns — One Gesture at a Time

If you're curious about how artificial intelligence learns there's no better way to explore than to build something yourself using Google's Teachable Machine.

My Mini Project: Gesture-Based Controls

Recently, I created a gesture-based game controller using this tool — no coding needed.

✨ My idea? Train an image classifier to recognize 3 hand gestures via webcam:

• 👍 Thumbs Up = “Go”

• ✋ Open Palm = “Stop”

• ✊ Fist = “Jump”

  
  

Using just my laptop webcam and Teachable Machine's image project, I trained a model to recognize these gestures. I took about 60–80 images per class under different lighting and angles, then exported the model to run directly in a browser. Within minutes, I had a working controller that could be used to trigger events in a simple game.

Seeing How AI Learns

The power of this tool wasn’t just in creating a model—it was in seeing how AI actually learns. The model hit ~96% accuracy and responded in real-time. But the real learning came from its limitations. Here are three core lessons:

1. AI Learns Patterns, Not Meaning

   The model didn’t "understand" a thumbs-up. It learned to associate the pattern of pixels that look like a thumbs-up with the label "Go." Without clear and consistent labels, there’s no intelligence at all.

   
   

2. Diversity = Strength

   My first model failed in different lighting. Why? Because I only trained under one condition. AI generalizes only as far as its training data allows. To improve the model, I added more images with shadows, different hand angles, and varied backgrounds.

   
   

3. Real-World Smoothness Requires Design

   When switching quickly between gestures, the model sometimes flickered between states. This highlighted a key design need: adding a transition buffer or smoothing logic. It's a valuable reminder that good AI products rely on more than just training data—they require thoughtful human-centered design.

Why You Should Try This As Well

You don’t need a PhD or Python skills. Teachable Machine lets anyone train a small-scale AI model in under an hour. It’s an excellent way to understand:

• Supervised learning

• Model accuracy vs generalization

• Why clean, diverse data matters

• The limits of AI in noisy environments

It’s also an empowering entry point for educators, creatives, kids, and anyone curious about what AI actually does.

If you would like a step by step guide you can keep reading or download it here in AI Experiments:

Teachable Machines Gesture Training Guide.pdf

Download PDF • 4.57MB

Teachable Machine Gesture Training Guide

Want to create your own gesture-controlled AI experiment using Google’s Teachable Machine? This step-by-step guide will walk you through the process of building a webcam-based image classifier that can recognize hand gestures like “thumbs up,” “open palm,” and “fist.”

🎯 Objective

Train a machine learning model using images of different hand gestures and use it to trigger events in a game or interactive application. No coding experience required!

🧰 What You’ll Need

• A laptop or desktop computer with a webcam

• Access to the internet

• Google Chrome or any updated browser

• A quiet, well-lit space for training images

• Optional: An account to save your project on Google Drive

🪜 Step-by-Step Instructions

1. Go to Teachable Machine

   Visit: https://teachablemachine.withgoogle.com/

   Click Get Started > Choose Image Project > Select Standard Image Model.

   
   

2. Set Up Your Classes

   Click Add a Class to create three classes:

   • Class 1: Thumbs Up (label it "Go")

   • Class 2: Open Palm (label it "Stop")

   • Class 3: Fist (label it "Jump")

     
     

3. Collect Your Training Data

   For each gesture:

   • Use your webcam to capture 60–80 images per class

   • Move your hand to slightly different positions, distances, and angles

   • Vary lighting conditions — turn lights on/off or shift location

   • Avoid any background clutter that might confuse the model

     
     

4. Train Your Model

   Once you’ve collected enough images:

   • Click the Train Model button

   • Wait while the model processes the training data (usually under 1 minute)

   • Once complete, you’ll see the live preview showing classification results in real-time

     
     

5. Export and Use the Model

   Once you’ve trained your model, here’s how to export and use it:

   
   

Option A: Use it in a Web Application

• Click Export Model

• Choose Upload (TensorFlow.js)

• This uploads the model to Google Cloud and gives you a model URL

• Use this URL in your JavaScript project following Google’s sample code or integrate it into a game prototype

Option B: Run Locally (Offline)

• Click Download to get a zip file

• Unzip to get model.json, metadata.json, and weights.bin

• Use TensorFlow.js to load the model in your browser or Node.js environment

Optional: Use the Live Preview

• Use the built-in preview pane to test the model with your webcam

• Watch real-time predictions and confidence scores

  
  

Connect to Game Logic (JavaScript Example):

if (prediction === "Go") {

    character.moveForward();

} else if (prediction === "Stop") {

    character.stop();

} else if (prediction === "Jump") {

    character.jump();

}

• Add debounce logic or smoothing for better UX

6. Improve Your Model

   • Add more diverse training images

   • Train in multiple environments

   • Smooth gesture transitions by filtering outputs over time

     
     

7. Test and Share

   • Try it in a simple browser game

   • Record your results

   • Share it on platforms like Glitch or GitHub Pages

🚀 Tips for Success

• Keep backgrounds neutral during training

• Avoid overly fast hand motions

• Use consistent gesture presentation