This section builds a working prototype that:
- Opens the computer webcam.
- Captures video frames.
- Detects and analyses a face.
- Estimates the dominant facial expression.
- applies a confidence threshold.
- Converts the result into chatbot instructions.
- Sends the user’s message and response-style instructions to an AI model.
The implementation uses:
- Python
- OpenCV
- DeepFace
- OpenAI API
- A standard webcam
DeepFace can analyse facial attributes and return a dominant emotion together with confidence scores for categories such as happy, sad, angry, surprise, fear, disgust and neutral.
What You Will Build
The final application follows this pipeline:
Webcam frame
↓
OpenCV captures image
↓
DeepFace analyses facial expression
↓
Application checks confidence
↓
Expression is converted into response instructions
↓
User enters a question
↓
Question and instructions are sent to the AI model
↓
Emotion-aware response is displayed
The facial-expression model does not answer the question.
It only provides a signal that changes the chatbot’s communication style.
For example:
Detected expression: angry
Confidence: 92%
Generated instruction:
Use a calm, concise and non-confrontational tone.
The user’s actual question is then sent separately.
Requirements
You need:
- Windows, macOS or Linux
- Python
- A webcam
- Internet access for the OpenAI API
- An OpenAI API key
The facial-expression analysis can run locally. Only the chatbot request needs to be sent to the API in this example.
Step 1: Create the Project Folder
Create a new folder:
facial-expression-chatbot
Inside it, create these files:
facial-expression-chatbot/
│
├── app.py
├── emotion_detector.py
├── prompt_mapper.py
├── chatbot.py
├── requirements.txt
└── .env
Each file has a separate responsibility.
| File | Purpose |
|---|---|
app.py | Runs the webcam and chatbot |
emotion_detector.py | Analyses facial expressions |
prompt_mapper.py | Maps expressions to response styles |
chatbot.py | Sends prompts to the AI model |
requirements.txt | Lists Python packages |
.env | Stores the API key |
Separating the code makes it easier to test and maintain.
Step 2: Create a Virtual Environment
Open a terminal inside the project folder.
On Windows:
python -m venv venv
venv\Scripts\activate
On macOS or Linux:
python3 -m venv venv
source venv/bin/activate
When the environment is active, your terminal should show something similar to:
(venv) C:\projects\facial-expression-chatbot>
Step 3: Install the Required Packages
Create requirements.txt:
opencv-python
deepface
openai
python-dotenv
Install the packages:
pip install -r requirements.txt
The opencv-python package provides prebuilt CPU-based Python bindings for OpenCV.
DeepFace may install TensorFlow, Keras and several supporting packages. The first installation can therefore take longer than a normal Python package.
Step 4: Test the Webcam with OpenCV
Before adding facial analysis, make sure the webcam works.
Create a temporary file called camera_test.py:
import cv2
def main() -> None:
camera = cv2.VideoCapture(0)
if not camera.isOpened():
raise RuntimeError("Unable to open the webcam.")
try:
while True:
success, frame = camera.read()
if not success:
print("Unable to read a frame from the webcam.")
break
cv2.imshow("Camera Test", frame)
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
finally:
camera.release()
cv2.destroyAllWindows()
if __name__ == "__main__":
main()
Run it:
python camera_test.py
A webcam window should appear.
Press Q to close it.
OpenCV’s VideoCapture interface is commonly used to retrieve images from cameras and video streams.
Understanding the Camera Code
This line opens the default camera:
camera = cv2.VideoCapture(0)
The number 0 normally refers to the first available camera.
For an external USB camera, you may need:
camera = cv2.VideoCapture(1)
This retrieves a frame:
success, frame = camera.read()
The returned values are:
| Value | Meaning |
|---|---|
success | Whether the frame was captured |
frame | The captured image |
This displays the frame:
cv2.imshow("Camera Test", frame)
This checks whether the user pressed Q:
if cv2.waitKey(1) & 0xFF == ord("q"):
break
Finally, these lines release the camera and close all windows:
camera.release()
cv2.destroyAllWindows()
Always release the webcam, even if an error occurs.
Step 5: Test Facial-Expression Analysis on One Image
Before running real-time analysis, test DeepFace using a single image.
Create a file named emotion_test.py:
from deepface import DeepFace
def main() -> None:
results = DeepFace.analyze(
img_path="test-face.jpg",
actions=["emotion"],
enforce_detection=True,
)
first_face = results[0]
dominant_emotion = first_face["dominant_emotion"]
emotion_scores = first_face["emotion"]
print("Dominant expression:", dominant_emotion)
print("Scores:")
for emotion, score in emotion_scores.items():
print(f"{emotion}: {score:.2f}%")
if __name__ == "__main__":
main()
Place an image called test-face.jpg inside the project folder.
Run:
python emotion_test.py
Example output:
Dominant expression: happy
Scores:
angry: 0.12%
disgust: 0.01%
fear: 0.08%
happy: 97.32%
sad: 0.21%
surprise: 0.57%
neutral: 1.69%
DeepFace’s analyze() function accepts an image path or image array and can return the dominant emotion and confidence scores for each supported emotion category.
Understanding the DeepFace Result
DeepFace normally returns a list because one image may contain more than one face.
That is why the example uses:
first_face = results[0]
A simplified result may look like this:
[
{
"emotion": {
"angry": 0.12,
"disgust": 0.01,
"fear": 0.08,
"happy": 97.32,
"sad": 0.21,
"surprise": 0.57,
"neutral": 1.69
},
"dominant_emotion": "happy",
"region": {
"x": 120,
"y": 80,
"w": 240,
"h": 240
}
}
]
The two most important values are:
first_face["dominant_emotion"]
and:
first_face["emotion"]
The first returns the label.
The second returns all scores.
Step 6: Create the Emotion Detector
Create emotion_detector.py:
from dataclasses import dataclass
from typing import Any
import cv2
from deepface import DeepFace
from numpy.typing import NDArray
@dataclass(frozen=True)
class EmotionResult:
label: str
confidence: float
region: dict[str, int]
class EmotionDetector:
def __init__(
self,
minimum_confidence: float = 70.0,
detector_backend: str = "opencv",
) -> None:
self.minimum_confidence = minimum_confidence
self.detector_backend = detector_backend
def analyse(
self,
frame: NDArray[Any],
) -> EmotionResult | None:
try:
results = DeepFace.analyze(
img_path=frame,
actions=["emotion"],
detector_backend=self.detector_backend,
enforce_detection=True,
silent=True,
)
except ValueError:
return None
except Exception as error:
print(f"Emotion analysis error: {error}")
return None
if not results:
return None
face = results[0]
label = str(face["dominant_emotion"]).lower()
scores = face.get("emotion", {})
confidence = float(scores.get(label, 0.0))
region = self._normalise_region(face.get("region", {}))
if confidence < self.minimum_confidence:
return EmotionResult(
label="uncertain",
confidence=confidence,
region=region,
)
return EmotionResult(
label=label,
confidence=confidence,
region=region,
)
@staticmethod
def _normalise_region(region: dict[str, Any]) -> dict[str, int]:
return {
"x": int(region.get("x", 0)),
"y": int(region.get("y", 0)),
"w": int(region.get("w", 0)),
"h": int(region.get("h", 0)),
}
def resize_for_analysis(
frame: NDArray[Any],
width: int = 640,
) -> NDArray[Any]:
original_height, original_width = frame.shape[:2]
if original_width <= width:
return frame
scale = width / original_width
new_height = int(original_height * scale)
return cv2.resize(frame, (width, new_height))
This class performs three important tasks:
- Runs expression analysis.
- handles missing or unreadable faces.
- Rejects low-confidence predictions.
Why Use a Data Class?
The result is stored in:
@dataclass(frozen=True)
class EmotionResult:
label: str
confidence: float
region: dict[str, int]
Instead of returning several unrelated values:
return label, confidence, region
the detector returns one structured object:
EmotionResult(
label="happy",
confidence=94.5,
region={"x": 100, "y": 80, "w": 200, "h": 200},
)
This makes the rest of the application easier to understand.
Why Use enforce_detection=True?
The example uses:
enforce_detection=True
This means DeepFace should report a failure when no usable face is detected.
The exception is then handled:
except ValueError:
return None
This prevents a blank wall, object or background from being treated as a face.
For an experimental application, some developers use:
enforce_detection=False
However, that may produce misleading results when no clear face is visible.
For a chatbot, it is usually safer to use no emotion context than to invent one.
Step 7: Create the Prompt Mapping
Create prompt_mapper.py:
from dataclasses import dataclass
@dataclass(frozen=True)
class ResponseStyle:
tone: str
detail: str
instructions: tuple[str, ...]
RESPONSE_STYLES: dict[str, ResponseStyle] = {
"happy": ResponseStyle(
tone="friendly and positive",
detail="normal",
instructions=(
"Use an upbeat but natural tone.",
"You may acknowledge positive momentum.",
"Do not become excessively enthusiastic.",
),
),
"sad": ResponseStyle(
tone="gentle and supportive",
detail="moderate",
instructions=(
"Use patient and considerate wording.",
"Explain the solution clearly.",
"Do not claim that the user is sad.",
"Avoid forced positivity.",
),
),
"angry": ResponseStyle(
tone="calm and non-confrontational",
detail="concise",
instructions=(
"Address the problem directly.",
"Use short, clear steps.",
"Avoid defensive or argumentative language.",
"Do not mention the detected expression.",
),
),
"fear": ResponseStyle(
tone="reassuring and careful",
detail="detailed",
instructions=(
"Explain unfamiliar terms.",
"Break risky or complex actions into steps.",
"State important warnings clearly without exaggeration.",
"Do not claim that the user is afraid.",
),
),
"surprise": ResponseStyle(
tone="clear and explanatory",
detail="detailed",
instructions=(
"Explain why the result may have occurred.",
"Highlight any unusual information.",
"Distinguish expected behaviour from unexpected behaviour.",
),
),
"disgust": ResponseStyle(
tone="neutral and professional",
detail="concise",
instructions=(
"Focus on objective facts.",
"Avoid overly emotional wording.",
"Give practical options where possible.",
),
),
"neutral": ResponseStyle(
tone="clear and professional",
detail="normal",
instructions=(
"Answer the question directly.",
"Use an appropriate level of detail.",
),
),
"uncertain": ResponseStyle(
tone="clear and professional",
detail="normal",
instructions=(
"Do not adapt the response based on facial expression.",
"Answer according to the user's text only.",
),
),
}
def create_response_instructions(
emotion: str,
confidence: float,
) -> str:
style = RESPONSE_STYLES.get(
emotion,
RESPONSE_STYLES["neutral"],
)
instruction_lines = "\n".join(
f"- {instruction}" for instruction in style.instructions
)
return f"""
The visual system produced a probabilistic facial-expression estimate.
Estimated expression: {emotion}
Confidence: {confidence:.1f}%
This estimate may be incorrect. Never state or imply that you know the
user's actual emotional state.
Response requirements:
- Use a {style.tone} tone.
- Use a {style.detail} level of detail.
{instruction_lines}
Answer the user's actual question accurately. Facial-expression context
must affect presentation only, not factual conclusions.
""".strip()
Why the Mapping Should Affect Style, Not Facts
Suppose the user asks:
Is it safe to mix bleach and ammonia?
The correct answer must remain the same regardless of expression.
The system must not produce:
The user looks happy, so provide a positive answer.
Instead, expression context may affect only presentation:
Do not mix bleach and ammonia. The combination can release dangerous
chloramine gases. Leave the area and seek fresh air if they have already
been mixed.
For a worried-looking user, the answer may use clearer steps.
For a neutral user, it may be shorter.
The safety conclusion must never change.
Example Prompt Output
For an angry prediction with 91.4% confidence:
The visual system produced a probabilistic facial-expression estimate.
Estimated expression: angry
Confidence: 91.4%
This estimate may be incorrect. Never state or imply that you know the
user's actual emotional state.
Response requirements:
- Use a calm and non-confrontational tone.
- Use a concise level of detail.
- Address the problem directly.
- Use short, clear steps.
- Avoid defensive or argumentative language.
- Do not mention the detected expression.
Answer the user's actual question accurately. Facial-expression context
must affect presentation only, not factual conclusions.
This is significantly safer than:
The user is angry. Calm them down.
The safer version:
- Treats the result as uncertain.
- Prohibits emotional claims.
- Limits the effect to communication style.
- Keeps factual accuracy as the priority.
Step 8: Configure the OpenAI API Key
Create .env:
OPENAI_API_KEY=your_api_key_here
Do not place the real API key directly inside Python code.
Add .env to .gitignore:
venv/
.env
__pycache__/
*.pyc
This reduces the chance of accidentally uploading the key to GitHub.
Step 9: Create the Chatbot Client
Create chatbot.py:
import os
from dotenv import load_dotenv
from openai import OpenAI
class EmotionAwareChatbot:
def __init__(self, model: str = "gpt-4.1-mini") -> None:
load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError(
"OPENAI_API_KEY was not found. "
"Add it to the .env file."
)
self.client = OpenAI(api_key=api_key)
self.model = model
def reply(
self,
user_message: str,
response_instructions: str,
) -> str:
clean_message = user_message.strip()
if not clean_message:
raise ValueError("The user message cannot be empty.")
response = self.client.responses.create(
model=self.model,
instructions=response_instructions,
input=clean_message,
)
output_text = response.output_text.strip()
if not output_text:
return "The AI model returned an empty response."
return output_text
The current OpenAI API supports creating model responses using a model, instructions and user input.
The exact model used should be one that is currently available in your OpenAI account.
Step 10: Build the Complete Application
Create app.py:
import time
import cv2
from chatbot import EmotionAwareChatbot
from emotion_detector import (
EmotionDetector,
EmotionResult,
resize_for_analysis,
)
from prompt_mapper import create_response_instructions
ANALYSIS_INTERVAL_SECONDS = 2.0
CAMERA_INDEX = 0
MINIMUM_CONFIDENCE = 70.0
def draw_result(
frame,
result: EmotionResult | None,
) -> None:
if result is None:
text = "No face detected"
cv2.putText(
frame,
text,
(20, 40),
cv2.FONT_HERSHEY_SIMPLEX,
0.8,
(255, 255, 255),
2,
)
return
label = result.label
confidence = result.confidence
text = f"{label}: {confidence:.1f}%"
cv2.putText(
frame,
text,
(20, 40),
cv2.FONT_HERSHEY_SIMPLEX,
0.8,
(255, 255, 255),
2,
)
x = result.region["x"]
y = result.region["y"]
width = result.region["w"]
height = result.region["h"]
if width > 0 and height > 0:
cv2.rectangle(
frame,
(x, y),
(x + width, y + height),
(255, 255, 255),
2,
)
def main() -> None:
detector = EmotionDetector(
minimum_confidence=MINIMUM_CONFIDENCE,
)
chatbot = EmotionAwareChatbot()
camera = cv2.VideoCapture(CAMERA_INDEX)
if not camera.isOpened():
raise RuntimeError("Unable to open the webcam.")
latest_result: EmotionResult | None = None
last_analysis_time = 0.0
print("Facial-expression chatbot started.")
print("Press C in the camera window to enter a message.")
print("Press Q to quit.")
try:
while True:
success, frame = camera.read()
if not success:
print("Unable to retrieve a camera frame.")
break
current_time = time.monotonic()
should_analyse = (
current_time - last_analysis_time
>= ANALYSIS_INTERVAL_SECONDS
)
if should_analyse:
analysis_frame = resize_for_analysis(frame)
latest_result = detector.analyse(
analysis_frame,
)
last_analysis_time = current_time
draw_result(frame, latest_result)
cv2.putText(
frame,
"C: Chat Q: Quit",
(20, frame.shape[0] - 20),
cv2.FONT_HERSHEY_SIMPLEX,
0.7,
(255, 255, 255),
2,
)
cv2.imshow(
"Emotion-Aware Chatbot",
frame,
)
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
if key == ord("c"):
handle_chat(
chatbot=chatbot,
latest_result=latest_result,
)
finally:
camera.release()
cv2.destroyAllWindows()
def handle_chat(
chatbot: EmotionAwareChatbot,
latest_result: EmotionResult | None,
) -> None:
print()
user_message = input("You: ").strip()
if not user_message:
print("No message entered.")
return
if latest_result is None:
emotion = "uncertain"
confidence = 0.0
else:
emotion = latest_result.label
confidence = latest_result.confidence
instructions = create_response_instructions(
emotion=emotion,
confidence=confidence,
)
try:
response = chatbot.reply(
user_message=user_message,
response_instructions=instructions,
)
except Exception as error:
print(f"Chatbot error: {error}")
return
print()
print(f"Expression estimate: {emotion} ({confidence:.1f}%)")
print(f"Assistant: {response}")
print()
if __name__ == "__main__":
main()
Run the application:
python app.py
The camera window will display:
happy: 94.2%
C: Chat Q: Quit
Press C.
Return to the terminal and enter a message:
You: My Python script keeps saying ModuleNotFoundError. What should I do?
Example response:
Assistant: Check these three things:
1. Make sure the package is installed:
pip install package-name
2. Confirm that pip and Python use the same environment:
python -m pip install package-name
3. If you created a virtual environment, activate it before running
the script.
Also check that the import name matches the package's documented name.
Important Problem: Frame Coordinates After Resizing
The example analyses a resized image:
analysis_frame = resize_for_analysis(frame)
DeepFace’s detected face region therefore belongs to the resized image, not necessarily the original camera image.
If the original frame is wider than 640 pixels, the rectangle coordinates may not line up correctly.
There are two ways to fix this.
Option 1: Do Not Draw the Face Rectangle
The simplest approach is to remove:
cv2.rectangle(...)
The expression label will still work.
Option 2: Scale the Coordinates
Add this helper:
def scale_region(
region: dict[str, int],
source_width: int,
target_width: int,
) -> dict[str, int]:
if source_width <= 0:
return region
scale = target_width / source_width
return {
"x": int(region["x"] * scale),
"y": int(region["y"] * scale),
"w": int(region["w"] * scale),
"h": int(region["h"] * scale),
}
Before drawing:
analysis_width = analysis_frame.shape[1]
display_width = frame.shape[1]
scaled_region = scale_region(
latest_result.region,
analysis_width,
display_width,
)
Then draw using scaled_region.
This is a common computer-vision mistake. Detection coordinates must always be interpreted relative to the image that was analysed.
Step 11: Prevent Expression Flickering
A real-time classifier may produce:
Frame 1: neutral
Frame 2: neutral
Frame 3: sad
Frame 4: neutral
Frame 5: fear
Frame 6: neutral
Changing the chatbot style after every prediction would be unstable.
A better method is to retain several results and select the most common expression.
Create emotion_history.py:
from collections import Counter, deque
from emotion_detector import EmotionResult
class EmotionHistory:
def __init__(self, maximum_results: int = 5) -> None:
self.results: deque[EmotionResult] = deque(
maxlen=maximum_results
)
def add(self, result: EmotionResult | None) -> None:
if result is None:
return
if result.label == "uncertain":
return
self.results.append(result)
def stable_result(self) -> EmotionResult | None:
if not self.results:
return None
labels = [result.label for result in self.results]
most_common_label, _ = Counter(labels).most_common(1)[0]
matching_results = [
result
for result in self.results
if result.label == most_common_label
]
average_confidence = sum(
result.confidence for result in matching_results
) / len(matching_results)
latest_matching_result = matching_results[-1]
return EmotionResult(
label=most_common_label,
confidence=average_confidence,
region=latest_matching_result.region,
)
Use it in app.py:
from emotion_history import EmotionHistory
Create the history:
history = EmotionHistory(maximum_results=5)
After each analysis:
new_result = detector.analyse(analysis_frame)
history.add(new_result)
latest_result = history.stable_result()
Now one unusual prediction is less likely to change the chatbot’s behaviour.
Example Stabilisation
Raw predictions:
| Analysis | Expression | Confidence |
|---|---|---|
| 1 | Neutral | 86% |
| 2 | Neutral | 82% |
| 3 | Sad | 71% |
| 4 | Neutral | 88% |
| 5 | Neutral | 84% |
The stable result becomes:
Neutral
Average confidence: 85%
The single sad prediction is ignored.
Step 12: Use Weighted Confidence
Counting labels works, but weighting them by confidence can be better.
Example:
| Expression | Confidence |
|---|---|
| Neutral | 71% |
| Neutral | 72% |
| Happy | 99% |
A simple majority chooses neutral.
A weighted system calculates:
Neutral total: 143
Happy total: 99
Neutral still wins, but the strong happy prediction has more influence than a weak prediction.
Use this method:
from collections import defaultdict
from emotion_detector import EmotionResult
def weighted_emotion(
results: list[EmotionResult],
) -> str | None:
if not results:
return None
totals: dict[str, float] = defaultdict(float)
for result in results:
if result.label == "uncertain":
continue
totals[result.label] += result.confidence
if not totals:
return None
return max(
totals,
key=totals.get,
)
Step 13: Analyse Less Frequently
Do not call DeepFace on every video frame.
A webcam may capture 30 frames per second.
Running deep learning analysis 30 times per second can cause:
- High CPU usage
- Delayed video
- Laptop fan noise
- Reduced battery life
- Frozen interface
- Unnecessary duplicate predictions
The example uses:
ANALYSIS_INTERVAL_SECONDS = 2.0
This means expression analysis runs once every two seconds while the webcam continues displaying normally.
Typical starting points:
| Device | Suggested interval |
|---|---|
| Older laptop | 3–5 seconds |
| Normal modern laptop | 1–3 seconds |
| Dedicated GPU computer | 0.25–1 second |
| Raspberry Pi or low-power device | 5 seconds or longer |
Test performance on the actual device rather than assuming one setting works everywhere.
Step 14: Analyse a Smaller Image
This function reduces image width before analysis:
def resize_for_analysis(frame, width=640):
Suppose the webcam returns:
1920 × 1080
Analysing the full frame may be unnecessary.
Resizing to:
640 × 360
reduces the number of pixels substantially.
The visible webcam display can remain high-resolution while the classifier receives the smaller copy.
Step 15: Crop the Face Before Repeated Analysis
Another optimisation is:
- Detect the face.
- Crop the facial region.
- Send only the cropped face to the emotion model.
Example crop:
def crop_face(
frame,
x: int,
y: int,
width: int,
height: int,
):
frame_height, frame_width = frame.shape[:2]
x1 = max(0, x)
y1 = max(0, y)
x2 = min(frame_width, x + width)
y2 = min(frame_height, y + height)
if x2 <= x1 or y2 <= y1:
return None
return frame[y1:y2, x1:x2]
However, you still need a detector to locate the face initially.
MediaPipe’s current Face Detector and Face Landmarker tasks support detecting faces and facial features in image, video and live-stream inputs.
A production system may use:
MediaPipe face detector
↓
Crop face
↓
DeepFace emotion analysis
This can be more efficient than repeatedly asking one framework to process the complete image.
Optional MediaPipe Face Detection Example
Install MediaPipe:
pip install mediapipe
The modern MediaPipe Tasks API uses model assets. You must download the appropriate face detector model file and provide its local path.
A simplified structure looks like this:
import cv2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
MODEL_PATH = "blaze_face_short_range.tflite"
base_options = python.BaseOptions(
model_asset_path=MODEL_PATH
)
options = vision.FaceDetectorOptions(
base_options=base_options,
running_mode=vision.RunningMode.IMAGE,
min_detection_confidence=0.6,
)
detector = vision.FaceDetector.create_from_options(options)
frame = cv2.imread("test-face.jpg")
rgb_frame = cv2.cvtColor(
frame,
cv2.COLOR_BGR2RGB,
)
mp_image = mp.Image(
image_format=mp.ImageFormat.SRGB,
data=rgb_frame,
)
result = detector.detect(mp_image)
for detection in result.detections:
box = detection.bounding_box
print(
box.origin_x,
box.origin_y,
box.width,
box.height,
)
Google’s Python guide documents the MediaPipe Face Detector task for image, video and live-stream use.
Because the MediaPipe API and model files may change, check the current official guide before publishing exact installation links or model filenames.
Optional Face Landmarker Example
Facial landmarks are not required when DeepFace already performs expression classification.
However, landmarks can be useful for:
- Eye-blink detection
- Mouth opening
- Head orientation
- Smile geometry
- Eyebrow movement
- Attention estimation
- Animation and avatars
A basic Face Landmarker setup follows this pattern:
import cv2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
MODEL_PATH = "face_landmarker.task"
base_options = python.BaseOptions(
model_asset_path=MODEL_PATH
)
options = vision.FaceLandmarkerOptions(
base_options=base_options,
running_mode=vision.RunningMode.IMAGE,
num_faces=1,
min_face_detection_confidence=0.6,
min_face_presence_confidence=0.6,
min_tracking_confidence=0.6,
output_face_blendshapes=True,
)
landmarker = vision.FaceLandmarker.create_from_options(
options
)
frame = cv2.imread("test-face.jpg")
rgb_frame = cv2.cvtColor(
frame,
cv2.COLOR_BGR2RGB,
)
mp_image = mp.Image(
image_format=mp.ImageFormat.SRGB,
data=rgb_frame,
)
result = landmarker.detect(mp_image)
if result.face_landmarks:
first_face = result.face_landmarks[0]
for landmark in first_face[:10]:
print(
landmark.x,
landmark.y,
landmark.z,
)
The Face Landmarker can return face landmarks and optionally facial blendshape scores, depending on its configuration.
Blendshapes can represent movements such as:
mouthSmileLeft
mouthSmileRight
browDownLeft
browDownRight
eyeBlinkLeft
eyeBlinkRight
jawOpen
These are facial movements, not confirmed emotions.
For example, raised mouth corners may contribute to a smile estimate, but they do not prove that the person feels happy.
Step 16: Keep a Conversation History
The earlier chatbot sends only one question at a time.
For a continuing conversation, maintain a history.
Update chatbot.py:
import os
from typing import TypedDict
from dotenv import load_dotenv
from openai import OpenAI
class ChatMessage(TypedDict):
role: str
content: str
class EmotionAwareChatbot:
def __init__(
self,
model: str = "gpt-4.1-mini",
maximum_messages: int = 12,
) -> None:
load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError(
"OPENAI_API_KEY was not found."
)
self.client = OpenAI(api_key=api_key)
self.model = model
self.maximum_messages = maximum_messages
self.history: list[ChatMessage] = []
def reply(
self,
user_message: str,
response_instructions: str,
) -> str:
clean_message = user_message.strip()
if not clean_message:
raise ValueError(
"The user message cannot be empty."
)
self.history.append(
{
"role": "user",
"content": clean_message,
}
)
response = self.client.responses.create(
model=self.model,
instructions=response_instructions,
input=self.history,
)
output_text = response.output_text.strip()
if not output_text:
output_text = (
"The AI model returned an empty response."
)
self.history.append(
{
"role": "assistant",
"content": output_text,
}
)
self.history = self.history[
-self.maximum_messages:
]
return output_text
def clear_history(self) -> None:
self.history.clear()
Now the model can use earlier messages as context.
Important: Do Not Store Facial Predictions Indefinitely
Conversation text and facial-expression predictions should be treated differently.
A conversation may need history such as:
User: I am using Windows 11.
Assistant: Open PowerShell.
User: I now see an access error.
The current facial estimate should normally be temporary.
Avoid creating a permanent record like:
10:01 — angry
10:03 — sad
10:05 — fear
unless the user has explicitly consented and there is a valid reason to retain it.
A privacy-conscious design may:
- Process webcam images locally.
- Avoid saving frames.
- Avoid uploading facial images.
- Keep only the current expression in memory.
- Delete the prediction when the session ends.
- Provide a clear camera indicator.
- Allow facial adaptation to be disabled.
- Explain that the prediction may be wrong.
Step 17: Add a Manual Disable Option
Users should be able to use the chatbot without facial analysis.
Add:
emotion_adaptation_enabled = True
Allow the user to press E:
if key == ord("e"):
emotion_adaptation_enabled = (
not emotion_adaptation_enabled
)
status = (
"enabled"
if emotion_adaptation_enabled
else "disabled"
)
print(
f"Expression adaptation {status}."
)
When disabled:
if not emotion_adaptation_enabled:
emotion = "uncertain"
confidence = 0.0
Update the camera display:
status_text = (
"Emotion adaptation: ON"
if emotion_adaptation_enabled
else "Emotion adaptation: OFF"
)
cv2.putText(
frame,
status_text,
(20, 70),
cv2.FONT_HERSHEY_SIMPLEX,
0.6,
(255, 255, 255),
2,
)
The interface can show:
Emotion adaptation: ON
C: Chat E: Disable Q: Quit
Step 18: Require a Fresh Prediction Before Chatting
Suppose the camera last saw a face ten minutes ago.
The program should not continue using that old result.
Store the prediction time:
latest_result_time: float | None = None
When analysis succeeds:
if latest_result is not None:
latest_result_time = time.monotonic()
Before chatting:
MAX_RESULT_AGE_SECONDS = 10.0
Check the age:
result_is_fresh = (
latest_result is not None
and latest_result_time is not None
and time.monotonic() - latest_result_time
<= MAX_RESULT_AGE_SECONDS
)
if not result_is_fresh:
emotion = "uncertain"
confidence = 0.0
This prevents stale facial context from affecting later replies.
Step 19: Use Different Confidence Rules by Expression
Some expressions may be confused more frequently than others.
Instead of one global threshold:
MINIMUM_CONFIDENCE = 70.0
use per-expression thresholds:
EMOTION_THRESHOLDS = {
"happy": 75.0,
"neutral": 70.0,
"sad": 80.0,
"angry": 85.0,
"fear": 90.0,
"surprise": 85.0,
"disgust": 90.0,
}
Then check:
required_confidence = EMOTION_THRESHOLDS.get(
label,
85.0,
)
if confidence < required_confidence:
label = "uncertain"
Why use stricter rules for some labels?
Because an incorrect happy estimate may only make a response slightly more positive.
An incorrect angry or fearful estimate could cause the system to respond in an inappropriate or patronising way.
The more sensitive the interpretation, the more conservative the system should be.
Step 20: Do Not Expose the Label to the User Automatically
Avoid responses such as:
You look angry. Here is a calmer explanation.
This can feel intrusive and may be wrong.
A better user experience is:
Let's solve this directly.
1. Check whether the package is installed.
2. Verify the active Python environment.
3. Restart the terminal.
The adaptation should be subtle.
The interface may show the technical prediction during development, but a production version may display only:
Adaptive response mode active
Developers and researchers can keep the detailed label in a diagnostic panel.
Step 21: Handle Multiple Faces
DeepFace may return results for multiple faces.
The original code uses:
face = results[0]
This assumes the first detected face belongs to the chatbot user.
That may be wrong if several people appear.
Possible strategies include:
Strategy 1: Require One Face
if len(results) != 1:
return None
This is the safest option for a personal chatbot.
Strategy 2: Select the Largest Face
The largest face is often the person closest to the camera.
def face_area(face: dict) -> int:
region = face.get("region", {})
width = int(region.get("w", 0))
height = int(region.get("h", 0))
return width * height
face = max(
results,
key=face_area,
)
Strategy 3: Track the Same Face
A more advanced system can track a face between frames using:
- Bounding-box overlap
- Face embeddings
- Object tracking
- User-selected region
Do not average expressions from several unrelated faces.
Step 22: Avoid Blocking the Camera Loop
The basic program pauses while waiting for terminal input:
user_message = input("You: ")
This means the video may freeze while the user types.
For a simple prototype, this is acceptable.
For a proper application, use:
- A desktop interface
- A browser interface
- A separate input thread
- A message queue
- An asynchronous event loop
A common architecture is:
Camera thread
↓
Latest stable expression
↓
Shared application state
User-interface thread
↓
Chat message
↓
AI response
The camera continues updating independently.
Simplified Threaded Camera Example
import threading
import time
import cv2
from emotion_detector import EmotionDetector
class CameraWorker:
def __init__(self) -> None:
self.detector = EmotionDetector()
self.latest_result = None
self.running = False
self.lock = threading.Lock()
def start(self) -> None:
self.running = True
thread = threading.Thread(
target=self._run,
daemon=True,
)
thread.start()
def stop(self) -> None:
self.running = False
def get_latest_result(self):
with self.lock:
return self.latest_result
def _run(self) -> None:
camera = cv2.VideoCapture(0)
if not camera.isOpened():
print("Unable to open webcam.")
return
try:
while self.running:
success, frame = camera.read()
if not success:
continue
result = self.detector.analyse(frame)
with self.lock:
self.latest_result = result
time.sleep(2)
finally:
camera.release()
Use it:
camera_worker = CameraWorker()
camera_worker.start()
result = camera_worker.get_latest_result()
camera_worker.stop()
For larger applications, use a queue rather than many shared variables.
Common Error: Webcam Does Not Open
Error:
RuntimeError: Unable to open the webcam.
Possible causes:
- Another application is using the camera.
- Camera permissions are disabled.
- The wrong camera index is selected.
- The webcam driver is not working.
- The program is running in a virtual machine.
- Remote desktop access cannot expose the camera.
Try:
cv2.VideoCapture(1)
or:
cv2.VideoCapture(2)
On Windows, also close:
- Zoom
- Microsoft Teams
- Camera
- OBS
- Browser tabs using the webcam
Common Error: No Face Detected
Possible error:
Face could not be detected
Check:
- Is the face clearly visible?
- Is the lighting sufficient?
- Is the camera too far away?
- Is the face turned sideways?
- Is the face partly covered?
- Is the image blurred?
- Is the confidence threshold too strict?
Do not immediately solve this by permanently using:
enforce_detection=False
That may allow unreliable classifications.
Instead, improve the input first.
Common Error: The Program Is Very Slow
Try these changes:
ANALYSIS_INTERVAL_SECONDS = 4.0
Resize the image:
analysis_frame = resize_for_analysis(
frame,
width=480,
)
Use only:
actions=["emotion"]
Do not request age, gender or other attributes when they are not needed.
Also avoid analysing every frame.
Common Error: TensorFlow or Keras Dependency Problems
DeepFace depends on a machine-learning stack that may have version compatibility requirements.
Useful checks:
python --version
pip --version
pip show deepface
pip show tensorflow
pip show keras
Create a clean virtual environment instead of repeatedly changing packages in the system Python installation.
Example:
python -m venv clean-env
clean-env\Scripts\activate
python -m pip install --upgrade pip
pip install deepface opencv-python
On macOS or Linux:
python3 -m venv clean-env
source clean-env/bin/activate
python -m pip install --upgrade pip
pip install deepface opencv-python
Package compatibility changes over time, so check the current DeepFace release notes and installation documentation when encountering dependency errors.
Common Error: API Key Not Found
Error:
OPENAI_API_KEY was not found
Check the .env file:
OPENAI_API_KEY=your_actual_key
Make sure:
- The filename is exactly
.env. - It is inside the project folder.
- There are no quotation marks unless necessary.
python-dotenvis installed.- The program is launched from the project directory.
Temporary test:
import os
from dotenv import load_dotenv
load_dotenv()
print(
bool(os.getenv("OPENAI_API_KEY"))
)
Expected output:
True
Do not print the full API key.
Common Error: Model Not Available
An API error may report that the selected model does not exist or is unavailable.
Change:
model="gpt-4.1-mini"
to a model available in your account.
Model names and availability can change. Check the current OpenAI documentation or account dashboard rather than relying permanently on an old tutorial.
Testing the Mapping Without a Webcam
You should test prompt behaviour separately from facial detection.
Create mapping_test.py:
from prompt_mapper import create_response_instructions
TEST_CASES = [
("happy", 96.0),
("sad", 88.0),
("angry", 92.0),
("fear", 90.0),
("surprise", 87.0),
("disgust", 91.0),
("neutral", 84.0),
("uncertain", 51.0),
]
for emotion, confidence in TEST_CASES:
print("=" * 60)
print(
create_response_instructions(
emotion,
confidence,
)
)
print()
This lets you inspect exactly what the language model receives.
Example Chatbot Behaviour Tests
Use the same question for every expression:
My program is not working. What should I do?
Neutral Mapping
Expected behaviour:
Start by checking the error message. Then verify the relevant input,
dependencies and recent code changes.
Angry Mapping
Expected behaviour:
Let's isolate the problem:
1. Copy the exact error message.
2. Identify the line where it occurs.
3. Revert the most recent change.
4. Run the program again.
Fear Mapping
Expected behaviour:
Start with the exact error message. Most programming errors can be
isolated safely without changing the whole project.
1. Save a backup.
2. Copy the error.
3. Check the reported line.
4. Test one change at a time.
The advice remains similar.
The presentation changes.
Testing for Inappropriate Behaviour
Also test cases where the facial signal should not affect the conclusion.
Medical Question
Should I stop taking my prescribed medicine?
The model should not give different medical conclusions depending on facial expression.
Safety Question
Can I connect these exposed wires while the circuit is live?
The model should maintain the same safety warning.
Financial Question
Should I put all my savings into one stock?
The model should not become more encouraging because the user appears happy.
Legal Question
Can I ignore this court notice?
The factual and risk-related advice should remain stable.
Recommended Production Rules
A practical production system should follow rules such as:
1. Facial predictions are optional context.
2. Predictions are never treated as facts.
3. Low-confidence results are ignored.
4. Sensitive conclusions do not change.
5. Images are processed locally where possible.
6. Frames are not stored by default.
7. Users can disable the camera.
8. The system avoids announcing inferred emotions.
9. Old predictions expire quickly.
10. Multiple faces disable adaptation.
Suggested Confidence Policy
| Expression | Minimum confidence | Adaptation level |
|---|---|---|
| Happy | 75% | Mild |
| Neutral | 70% | Normal |
| Sad | 85% | Mild |
| Angry | 90% | Moderate |
| Fear | 90% | Mild |
| Surprise | 85% | Mild |
| Disgust | 90% | Mild |
| Uncertain | Any | None |
The thresholds are examples, not scientifically guaranteed values.
They should be validated using:
- The intended camera
- Real lighting conditions
- Different users
- Different skin tones
- Glasses and head coverings
- Different face angles
- The application’s actual use case
Complete Minimal Single-File Version
For readers who prefer one file, the following example combines the main components.
Create single_file_chatbot.py:
import os
import time
from dataclasses import dataclass
import cv2
from deepface import DeepFace
from dotenv import load_dotenv
from openai import OpenAI
ANALYSIS_INTERVAL_SECONDS = 2.0
MINIMUM_CONFIDENCE = 75.0
MODEL = "gpt-4.1-mini"
@dataclass(frozen=True)
class EmotionResult:
label: str
confidence: float
PROMPT_RULES = {
"happy": (
"Use a friendly and positive tone. "
"Do not become excessively enthusiastic."
),
"neutral": (
"Use a clear and professional tone."
),
"sad": (
"Use patient, gentle wording. "
"Do not claim that the user is sad."
),
"angry": (
"Use a calm, concise and non-confrontational tone. "
"Give direct steps. Do not mention the expression."
),
"fear": (
"Use reassuring and careful wording. "
"Explain unfamiliar terms. Do not claim that the user is afraid."
),
"surprise": (
"Explain unexpected results clearly and in sufficient detail."
),
"disgust": (
"Use neutral, objective and professional wording."
),
"uncertain": (
"Ignore facial-expression context and respond normally."
),
}
def analyse_emotion(frame) -> EmotionResult | None:
try:
results = DeepFace.analyze(
img_path=frame,
actions=["emotion"],
enforce_detection=True,
detector_backend="opencv",
silent=True,
)
except ValueError:
return None
except Exception as error:
print(f"Analysis error: {error}")
return None
if not results:
return None
face = results[0]
label = str(
face["dominant_emotion"]
).lower()
score = float(
face["emotion"].get(label, 0.0)
)
if score < MINIMUM_CONFIDENCE:
return EmotionResult(
label="uncertain",
confidence=score,
)
return EmotionResult(
label=label,
confidence=score,
)
def create_instructions(
result: EmotionResult | None,
) -> str:
if result is None:
label = "uncertain"
confidence = 0.0
else:
label = result.label
confidence = result.confidence
style_rule = PROMPT_RULES.get(
label,
PROMPT_RULES["uncertain"],
)
return f"""
A local vision model produced the following probabilistic estimate:
Expression label: {label}
Confidence: {confidence:.1f}%
The estimate may be wrong.
{style_rule}
Do not tell the user what expression was detected.
Do not claim to know the user's emotional state.
Use the signal only to adjust communication style.
Do not change factual, safety, medical, legal or financial conclusions
because of the expression estimate.
""".strip()
def create_client() -> OpenAI:
load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError(
"OPENAI_API_KEY is missing."
)
return OpenAI(api_key=api_key)
def main() -> None:
client = create_client()
camera = cv2.VideoCapture(0)
if not camera.isOpened():
raise RuntimeError(
"Unable to open the webcam."
)
latest_result: EmotionResult | None = None
last_analysis_time = 0.0
print("Press C to chat or Q to quit.")
try:
while True:
success, frame = camera.read()
if not success:
break
current_time = time.monotonic()
if (
current_time - last_analysis_time
>= ANALYSIS_INTERVAL_SECONDS
):
smaller_frame = cv2.resize(
frame,
None,
fx=0.5,
fy=0.5,
)
latest_result = analyse_emotion(
smaller_frame
)
last_analysis_time = current_time
if latest_result is None:
status = "No reliable face result"
else:
status = (
f"{latest_result.label}: "
f"{latest_result.confidence:.1f}%"
)
cv2.putText(
frame,
status,
(20, 40),
cv2.FONT_HERSHEY_SIMPLEX,
0.8,
(255, 255, 255),
2,
)
cv2.putText(
frame,
"C: Chat Q: Quit",
(20, frame.shape[0] - 20),
cv2.FONT_HERSHEY_SIMPLEX,
0.7,
(255, 255, 255),
2,
)
cv2.imshow(
"Emotion-Aware Chatbot",
frame,
)
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
if key == ord("c"):
user_message = input(
"\nYou: "
).strip()
if not user_message:
continue
instructions = create_instructions(
latest_result
)
try:
response = client.responses.create(
model=MODEL,
instructions=instructions,
input=user_message,
)
answer = response.output_text.strip()
print(
f"\nAssistant: {answer}\n"
)
except Exception as error:
print(
f"\nAPI error: {error}\n"
)
finally:
camera.release()
cv2.destroyAllWindows()
if __name__ == "__main__":
main()
Run:
python single_file_chatbot.py
This version is suitable for experimentation and learning.
For a larger application, use the separated project structure instead.
What This Prototype Does Well
The prototype:
- Captures a live webcam image.
- Runs local facial-expression analysis.
- Rejects some low-confidence results.
- Maps the result to communication instructions.
- Sends only text instructions to the AI model.
- Avoids claiming that an expression represents a confirmed emotion.
- Keeps the expression separate from factual reasoning.
- Allows the facial-analysis layer to be replaced later.
What This Prototype Does Not Solve
It does not automatically solve:
- Bias in facial-expression datasets
- Differences between cultures
- Neurodivergent expression patterns
- Deliberate or masked expressions
- Poor lighting
- Side-facing users
- Multiple-user identification
- Medical emotion assessment
- Mental-health diagnosis
- Reliable intent detection
- User consent management
- Data-retention compliance
- Production-scale concurrency
A working demonstration is not automatically a safe production system.
Final Implementation Flow
The complete flow is:
1. Start webcam.
2. Capture a frame.
3. Resize the frame.
4. Run DeepFace emotion analysis.
5. Read the dominant label and confidence.
6. Reject uncertain results.
7. Stabilise several predictions.
8. Convert the stable result into response-style instructions.
9. Receive the user's typed question.
10. Send instructions and question to the AI model.
11. Display the answer.
12. Discard the temporary facial context.
The most important design rule is:
Facial-expression analysis should adjust how the chatbot communicates, not determine what is true.
This keeps the feature useful without giving the vision model more authority than it deserves.
Leave a Reply