Smart Living Products
ISDN2001/2002: Second Year Design Project
AI-Powered Intelligent Ingredient Detection for the Modern Kitchen
Clarissa Lam
About the project
The project I am working on is a smart kitchen assistant that watches your cutting board and intelligently identifies ingredients using computer vision and multimodal AI.
It serves as the Ingredient Detection module of the larger RATATOUILLE system, enabling customized recipe generation and step-by-step guidance based on what you actually have.
Technical Approach
The system uses a hybrid computer vision architecture optimized for the Nvidia Jetson Nano:
YOLOv8 (Spotter & Assistant)
-
Detects the cutting board and fiducial markers
-
Draws bounding boxes
-
Crops the image to focus on the region of interest
LLaVA (Core Analysis Model via Ollama)
-
Performs zero-shot ingredient recognition
-
Provides semantic understanding and contextual descriptions
-
Differentiates between similar items (e.g., different fruits)
This hybrid approach combines YOLOv8’s speed and spatial precision with LLaVA’s powerful reasoning capabilities.
Key Technologies
1 Nvidia Jetson Nano
Edge computing platform for real-time processing
2 YOLOv8
Fast object detection for board and region identification
3 LLaVA + Ollama
Local multimodal vision-language model for ingredient understanding
4 Fiducial Markers
For accurate board calibration and spatial reference
5 Database Integration
Persistent storage of ingredient data