On Day 3, we’ll enhance the AI assistant by detecting facial expressions and mapping them to chat responses. This will allow the chatbot to react to emotions by changing its own avatar expressions.
1. Why Map Facial Expressions to Chatbot Reactions?
✅ Enhances Realism – The chatbot reacts visually to users’ emotions.
✅ Improves Engagement – The AI avatar appears more human-like.
✅ Enables Emotion-Based Responses – If the user looks sad, the chatbot can offer encouragement.
We’ll use:
🔹 MediaPipe Face Mesh – Detects facial landmarks and expressions.
🔹 TensorFlow.js – Processes real-time emotion detection.
🔹 Three.js or React Three Fiber – Maps expressions to the 3D avatar.
2. Installing Facial Expression Detection Dependencies
Step 1: Install TensorFlow.js & MediaPipe Face Mesh
npm install @tensorflow/tfjs @tensorflow-models/facemesh
Step 2: Install Expo Camera for Real-Time Video
expo install expo-camera
3. Detecting Facial Expressions from the User
Step 1: Create FaceExpressionDetector.js
Inside src/components/, create a new file:
import React, { useState, useEffect } from 'react';
import { View, Text, StyleSheet } from 'react-native';
import { Camera } from 'expo-camera';
import * as tf from '@tensorflow/tfjs';
import * as faceLandmarksDetection from '@tensorflow-models/face-landmarks-detection';
export default function FaceExpressionDetector({ onExpressionDetect }) {
const [hasPermission, setHasPermission] = useState(null);
const [expression, setExpression] = useState('');
useEffect(() => {
(async () => {
const { status } = await Camera.requestPermissionsAsync();
setHasPermission(status === 'granted');
})();
}, []);
useEffect(() => {
const loadModel = async () => {
await tf.ready();
const model = await faceLandmarksDetection.load(
faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
);
detectFace(model);
};
loadModel();
}, []);
const detectFace = async (model) => {
const predictions = await model.estimateFaces({ input: document.createElement('canvas') });
if (predictions.length > 0) {
const { mouthLeft, mouthRight, eyeLeft, eyeRight } = predictions[0].scaledMesh;
const smileWidth = Math.abs(mouthLeft[0] - mouthRight[0]);
const eyeDistance = Math.abs(eyeLeft[1] - eyeRight[1]);
if (smileWidth > 0.08) setExpression('smiling');
else if (eyeDistance < 0.02) setExpression('blinking');
else setExpression('neutral');
onExpressionDetect(expression);
}
requestAnimationFrame(() => detectFace(model));
};
if (hasPermission === null) return <View />;
if (hasPermission === false) return <Text>No access to camera</Text>;
return (
<View style={styles.container}>
<Camera style={styles.camera} type={Camera.Constants.Type.front} />
<Text style={styles.text}>Detected Expression: {expression}</Text>
</View>
);
}
const styles = StyleSheet.create({
container: { flex: 1 },
camera: { flex: 1 },
text: { fontSize: 18, fontWeight: 'bold', textAlign: 'center', marginTop: 20 },
});
4. Integrating Facial Expressions into the Chatbot
Modify ChatBot.js to receive facial expressions:
import FaceExpressionDetector from './FaceExpressionDetector';
const [userExpression, setUserExpression] = useState('neutral');
<FaceExpressionDetector onExpressionDetect={setUserExpression} />;
Modify sendMessage to adjust chatbot responses based on expressions:
const sendMessage = async (input) => {
if (!input.trim()) return;
const emotionContext = {
smiling: "I see you're happy! 😊",
blinking: "Are you tired? Maybe take a short break. 😴",
neutral: '',
};
const emotionResponse = emotionContext[userExpression] || '';
const response = await axios.post(
'https://api.openai.com/v1/chat/completions',
{
model: 'gpt-4',
messages: [{ role: 'user', content: input + ' ' + emotionResponse }],
},
{ headers: { Authorization: `Bearer ${OPENAI_API_KEY}` } }
);
setMessages([...messages, { text: response.data.choices[0].message.content, sender: 'bot' }]);
};
5. Mapping Expressions to the AI Avatar
Modify AvatarModel.js to change facial expressions dynamically:
function AvatarModel({ expression }) {
return (
<group>
<mesh>
<sphereGeometry args={[1, 32, 32]} />
<meshStandardMaterial color="orange" />
</mesh>
{/* Mouth expression */}
<mesh position={[0, -0.3, 1]} scale={[1, expression === 'smiling' ? 1.2 : 1, 1]}>
<boxGeometry args={[0.4, 0.2, 0.1]} />
<meshStandardMaterial color="red" />
</mesh>
{/* Eyes closing when blinking */}
<mesh position={[-0.3, 0.5, 1]} scale={[1, expression === 'blinking' ? 0.2 : 1, 1]}>
<sphereGeometry args={[0.1, 16, 16]} />
<meshStandardMaterial color="black" />
</mesh>
</group>
);
}
Modify AvatarRenderer.js to connect the avatar to expressions:
const [facialExpression, setFacialExpression] = useState('neutral');
<FaceExpressionDetector onExpressionDetect={setFacialExpression} />;
<AvatarModel expression={facialExpression} />;
6. Testing the Emotion-Responsive AI Avatar
Step 1: Start the App
expo start
Step 2: Test Different Expressions
- Smile → The chatbot detects happiness and responds positively.
- Blink → The chatbot suggests taking a break.
- Stay neutral → The chatbot responds normally.
Step 3: Verify Avatar Expressions
- Smile → Avatar widens mouth.
- Blink → Avatar’s eyes close briefly.
7. Optimizing Performance
✅ Reduce Processing Lag
- Process expressions every 3rd frame instead of every frame:
if (frameCount % 3 === 0) detectFace(model);
✅ Use WebGL Acceleration
tf.setBackend('webgl');
✅ Optimize Emotion Processing
const emotionThreshold = { smiling: 0.08, blinking: 0.02 };
8. Key Concepts Covered
✅ Detected user facial expressions using MediaPipe Face Mesh.
✅ Mapped expressions to chatbot responses.
✅ Integrated avatar facial movements based on user emotions.
9. Next Steps: Integrating AI-Based Speech Emotion Analysis
Tomorrow, we’ll:
🔹 Detect user emotions from voice tone.
🔹 Make the chatbot adjust responses based on vocal cues.
10. References & Learning Resources
11. SEO Keywords:
React Native AI avatars, real-time facial expression detection, MediaPipe face tracking, AI chatbot emotions, integrating facial recognition with chatbots.
