On Day 6, we’ll introduce gesture recognition into our AI-powered avatar app. This will allow users to wave to start a conversation, raise a hand to stop, or thumbs-up to confirm—making the experience even more natural and hands-free.
1. Why Add Gesture Control?
✅ Hands-Free Interaction – Users can control the avatar using simple hand signals.
✅ Enhanced User Experience – More intuitive than tapping buttons, especially in AI assistant apps.
✅ Combining Voice + Gesture – Builds a multi-modal AI system, improving accessibility.
We’ll use: 🔹 MediaPipe Hands API – For real-time hand tracking and gesture detection.
🔹 TensorFlow.js – To process hand landmarks for gestures.
🔹 React Three Fiber – To animate avatar reactions based on hand signals.
2. Installing Hand Tracking Dependencies
Step 1: Install TensorFlow.js & MediaPipe Hands
npm install @tensorflow/tfjs @tensorflow-models/handpose
Step 2: Install Expo Camera for Hand Detection
expo install expo-camera
3. Setting Up Gesture Detection
Step 1: Create GestureDetector.js
import React, { useState, useEffect } from 'react';
import { View, Text, StyleSheet } from 'react-native';
import { Camera } from 'expo-camera';
import * as tf from '@tensorflow/tfjs';
import * as handpose from '@tensorflow-models/handpose';
export default function GestureDetector({ onGestureRecognized }) {
const [hasPermission, setHasPermission] = useState(null);
const [gesture, setGesture] = useState('None');
useEffect(() => {
(async () => {
const { status } = await Camera.requestPermissionsAsync();
setHasPermission(status === 'granted');
await tf.ready();
})();
}, []);
useEffect(() => {
let model;
const loadModel = async () => {
model = await handpose.load();
};
loadModel();
const detectGesture = async (camera) => {
if (!model) return;
const predictions = await model.estimateHands(camera);
if (predictions.length > 0) {
const landmarks = predictions[0].landmarks;
const thumbTip = landmarks[4];
const indexTip = landmarks[8];
const palmBase = landmarks[0];
const thumbUp = thumbTip[1] < indexTip[1] && thumbTip[1] < palmBase[1];
const wavingHand = Math.abs(thumbTip[0] - palmBase[0]) > 80; // Horizontal spread
if (thumbUp) {
setGesture('Thumbs Up');
onGestureRecognized('thumbs_up');
} else if (wavingHand) {
setGesture('Waving');
onGestureRecognized('wave');
} else {
setGesture('None');
}
} else {
setGesture('None');
}
requestAnimationFrame(() => detectGesture(camera));
};
return () => clearInterval(detectGesture);
}, []);
if (hasPermission === null) return <View />;
if (hasPermission === false) return <Text>No access to camera</Text>;
return (
<View style={styles.container}>
<Text style={styles.text}>Detected Gesture: {gesture}</Text>
</View>
);
}
const styles = StyleSheet.create({
container: { padding: 10 },
text: { fontSize: 16 },
});
4. Integrating Gesture Control with the Avatar
Modify AvatarRenderer.js to respond to gestures:
import React, { useState } from 'react';
import { Canvas } from '@react-three/fiber';
import AvatarAnimation from './AvatarAnimation';
import GestureDetector from './GestureDetector';
export default function AvatarRenderer() {
const [gesture, setGesture] = useState('neutral');
const handleGestureRecognized = (detectedGesture) => {
if (detectedGesture === 'thumbs_up') setGesture('happy');
if (detectedGesture === 'wave') setGesture('waving');
};
return (
<>
<GestureDetector onGestureRecognized={handleGestureRecognized} />
<Canvas>
<ambientLight intensity={0.5} />
<directionalLight position={[0, 5, 5]} intensity={1} />
<AvatarAnimation emotion={gesture} />
</Canvas>
</>
);
}
5. Expanding Gesture-Based Commands
✅ Wave Gesture → Start Interaction
- When the user waves, the assistant greets them.
✅ Thumbs-Up Gesture → Confirm Action
- When the user gives a thumbs up, the assistant confirms their request.
✅ Palm Raise → Stop Interaction
- Future: Raise hand → Stop speaking.
Modify ChatBot.js:
const [gestureInput, setGestureInput] = useState('');
const handleGestureCommand = (gesture) => {
if (gesture === 'wave') {
setGestureInput('Hello!');
} else if (gesture === 'thumbs_up') {
setGestureInput('Yes, confirm that!');
}
};
<GestureDetector onGestureRecognized={handleGestureCommand} />;
6. Testing Gesture-Based Control
Step 1: Start the App
expo start
Step 2: Try These Gestures
| Gesture | Expected Avatar Response | Expected Chatbot Behavior |
|---|---|---|
| Wave | Avatar waves | Bot: “Hello! How can I help?” |
| Thumbs Up | Avatar nods/smiles | Bot: “Got it! Confirmed.” |
| Neutral | Avatar idle | Bot: Does nothing |
7. Improving Hand Gesture Recognition
✅ Adjust Sensitivity
- Fine-tune landmark distances based on testing:
const wavingThreshold = 60; // Adjust based on your hand-camera distance
✅ Track Hand Confidence Scores
- Use confidence to reduce false positives:
if (predictions[0].handInViewConfidence > 0.9) { /* Accept gesture */ }
✅ Add More Gestures
- Victory Sign (Peace) → “Good job!”
- Raised Palm → “Stop talking.”
- Fist → “Pause conversation.”
8. Key Concepts Covered
✅ Integrated hand gesture recognition using MediaPipe Hands API.
✅ Linked gestures to avatar animations like waving & nodding.
✅ Triggered chatbot commands using hand signals.
9. Next Steps: Personalizing Avatar Responses
Tomorrow, we’ll:
🔹 Customize chatbot replies based on user preferences.
🔹 Enable the avatar to remember names, greetings, or moods.
10. References & Learning Resources
11. SEO Keywords:
React Native hand tracking, AI avatars with gesture control, MediaPipe hand detection, TensorFlow.js handpose model, building voice + gesture-controlled assistants.
