8 ms·
I want to build an application that scans restaurant and café menus (PDFs, photos, webpages) to identify which items are vegetarian or vegan. Would this work fo
by vosper 3y ago
I want to build an application that scans restaurant and café menus (PDFs, photos, webpages) to identify which items are vegetarian or vegan. Would this work for that? If not, I would love to hear peoples ideas and suggestions.
- 7moritz7 3y agoThe non quantized models look relatively large, 800 MB. So you'd probably need to do inference on a server and somehow monetize that. Sounds difficult
- kolinko 3y agoWith vegan you can’t estimate it 100% from menu alone - because the sauce and other minor ingredients can be animal based. If you want to do it, using “plant based” is probably better than “vegan”, and it’s always good to make sure your users are aware that the mark can be wrong and they should double-check with the waiter. As for your question - I didn’t play with Donut, but ocr+gpt or multimodal gpt4 once released should handle this smoothly.
- ada1981 3y ago20+ years vegan here. You could combine ingredient search, looking for symbols the actually designate vegan as some places do, along with long/lat data to determine what the restaurant actually is and then check it with a database you maintain. So I could scan a menu, and then ask the owner or server about certain dishes, and then crowdsource an updated It would be great if there was a standard API for all restaurants that included all menu items, prices, ingredients, preparation and sourcing information. I could be maintained like a wiki I suppose, and restaurants could be incentivized by including their restaurants.
- vosper 3y agoYeah crowd-sourcing updates is the way I'd like to go. I'm hoping people will submit photos of menus, because lots of bakeries, cafés etc don't have much of an online presence or keep their menus up-to-date. Trying to solve the problem of scanning through menus for multiple restaurants to find something a vegan or vegetarian can eat, and instead just showing all the individual menu options in the area as a list.
- ada1981 3y agoOf course, anytime someone scans a menu you will have gps info available, so you can include it automatically
- Alifatisk 3y agoWe have a similar idea, but mine includes a few other categories!
- amelius 3y agoI bet in a year or so you don't need a specialized app for it, but you just ask your phone whatever you want to know about anything around you, including menus.
- Alifatisk 3y agoGoogle maps is pretty close to that, I can already find lots of places or restaurants based on what I want to eat. And now, thanks to crows sourcing, you can filter by a range of options. The only problem is that the data is closely walled by Google and you can only access it through their api. What I want to create is a tiny search engine that collects all menus (somehow collected from images) and let users find and filter what they like, and even get recommendations nearby!
- amelius 3y ago> The only problem is that the data is closely walled by Google and you can only access it through their api. Perhaps if iOS would allow an AI to access the screen pixels directly ...
- tuardoui 3y agoYou should look at LayoutLM models for a NER task. Then your pipeline should look like : - Identity the menu sub structure (title, item list ...) - Classify each item with 2 labels. The training process is not hard, but the data gathering / cleaning / labelling can be a little long.
- vosper 3y agoThanks! I haven't heard of LayoutLM but something that can understand structure from a few examples could be just what I need.