Leveraging artificial intelligence for automated menu extraction and allergen detection from restaurant websites
Jussila, Heli (2025)
2025
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025112630010
https://urn.fi/URN:NBN:fi:amk-2025112630010
Tiivistelmä
The purpose of the thesis is to explore how artificial intelligence can be used to automatically find menu and contact details from the restaurant website and extract allergens from the dishes.
The need for this kind of system arises from the increasing demand for digital food-related services and the difficulties collecting this information from websites that vary greatly in layout, structure, and language. The work aims to develop a system that could automate this process using recent AI technologies.
The system is based on an AI agent that visits restaurant website and locates and extracts relevant content using a combination of Playwright and a vision-language model (VLM). Playwright takes the screenshot of the page and that is sent to VLM. VLM interprets the screenshot and then sends the next action for Playwright to execute. This loop continues until the menu and contact details are found or the pre-defined maximum number of steps is used.
The first round of allergen detection includes finding the ingredients for the dishes and then the ingredients and screenshot of the menu is sent to VLM to predict the allergens and return them in structured format.
The need for this kind of system arises from the increasing demand for digital food-related services and the difficulties collecting this information from websites that vary greatly in layout, structure, and language. The work aims to develop a system that could automate this process using recent AI technologies.
The system is based on an AI agent that visits restaurant website and locates and extracts relevant content using a combination of Playwright and a vision-language model (VLM). Playwright takes the screenshot of the page and that is sent to VLM. VLM interprets the screenshot and then sends the next action for Playwright to execute. This loop continues until the menu and contact details are found or the pre-defined maximum number of steps is used.
The first round of allergen detection includes finding the ingredients for the dishes and then the ingredients and screenshot of the menu is sent to VLM to predict the allergens and return them in structured format.