4 ms·

AI Restaurant Menu with RAG

61 points by byyoung3 3y ago

Animats 3y ago
For a single menu, this is silly. As a search engine for Doordash or Uber Eats, though, it might lead to this guy being acquired.
- byyoung3 3y ago
  1m obo
- dkarras 3y ago
  I mean even as a single menu, I'm not a big fan of sifting through the menu of a restaurant, especially if I'm eating because I have to. If I could type (or even better speak) what I'm into and what I'm willing to spend and if there pops up the options, it would be a quality of life improvement for me.
jumploops 3y ago
Semantic search with vector embeddings is great... as long as you maintain an inverted index for traditional full-text search. As a user I sometimes want a "buffalo chicken quesadilla" not disparate buffalo wings and quesadilla recommendations. I believe Elastic has a mechanism that supports both? Haven't tested it yet.
- CoastalCoder 3y ago
  > As a user I sometimes want a "buffalo chicken quesadilla" not disparate buffalo wings and quesadilla recommendations. Curse you! I just started eating dinner, but now I want what you listed instead.
  - all2 3y ago
    Uhh, let's see here Roasted buffalo chicken thighs, smoked gouda, and sautéed onions, bell pepper (and jalapeño or other peor is optional, there's a smoked canned pepper -- I cannot remember the name -- that could be especially good here). This is pretty straight forward: 1. Place chicken thighs (bone in, skin on) on a roasting sheet, liberally salt. Roast in an oven at 400F for 30 to 40 minutes (start checking at the 30 minute mark). 2. While the chicken roasts, dice an onion, a red bell pepper, and a green bell pepper and sautee in oil of your choice (bonus points for using the drippings from the chicken thighs). Don't cook them all the way through. 3. Cook down canned black beans with a bay leaf, 1/4c of fat of your choice, salt (to taste), cumin (to taste). They're done about the time this concoction resembles a spreadable paste, about 10 minutes over medium heat. 4. Shred the thighs in a bowl, consume or set aside the skin. Add your buffalo sauce if choice. Bonus points if you made your own Buffalo sauce. 5. Heat a cast iron skillet over medium high heat, or a George foreman style press. I'm assuming these are non stick, add enough oil that there is a fine sheen. We don't want to fry the quesadillas (maybe next recipe). 6. Assemble your quesadillas directly in the pan, in this order: cheese (shredded or slice thin is fine), veg, meat, beans, cheese other tortilla. The last layer of cheese is important and will help glue the thing shut. If you're using a skillet, put a lid on it to tap heat and encourage the top cheese to start melting. 7. When the cheese on the bottom has melted and the tortilla is starting to brown, we flip. Get a plate and slide the quesadilla onto it. Then take the skillet and put it on top and then rotate both. If you're using the press, get a second plate and flip. 8. Wait for the new bottom side to brown and melt. Finish with cilantro, lime juice, and something approaching queso fresco. Greek yogurt, sour cream, goat cheese, anything like that will do. Enjoy! now I also want Buffalo chicken quesadillas [edit] Chipotle! It only took an hour to remember.
sebastiennight 3y ago
Aha interesting. I would guess it is missing a bit of nuance, eg a real word query would not be "appetizers with chicken" but something closer to "I'm waiting for my friends, not too hungry, what do you have that's spicy but not too filling? Oh, and I don't eat red meat". This would be much harder with basic vector database+RAG. You would at least need HyDE (hypothetical document embeddings) to act like a real waiter. I find your project very interesting though, and the walkthrough is nice!
- byyoung3 3y ago
  Agreed. Rag isn't ideal for every scenario. Putting the contents of the menu in the prompt could even be feasible with a smaller model.
- dartos 3y ago
  I feel like HyDE would just cause hallucinations. Imagine the generated documents based on that query was like “Our shepard’s pie is the perfect spicy meal for sharing with friends without being filling” (Obviously this is wrong, but it’s an example of a statistically possible document) Then the following similarly search would also be searching for shepherds pie. I don’t think throwing more LLMs at rag problems is the answer.
chadash 3y ago
> Physical menus are fine. They do the job. But if we're looking to actually improve upon them, current approaches miss the mark. They offer limited interactivity and flexibility. Sigh. Am I the only one who likes to hold a physical menu and browse it? I never go into a restaurant knowing what I’m gonna get… I like to see how the menu is themed and decide from there. The UI on a physical menu is a vast improvement over any digital menu I’ve seen.
- flextheruler 3y ago
  Yeah I’m not sure which restaurants would use something like this maybe food trucks or casual places that already do the QR pdf menus. The menu is a huge portion of setting the ambience and theme of a nice restaurant.
  - alwa 3y ago
    And beyond the setting and ambiance, it’s a huge portion of the restaurant’s value proposition: “Here are all the things we do and are proud of, what sounds best?” At the risk of being extra curmudgeonly, have you ever dined someplace where they wheel a dessert cart round? There’s kind of a lightweight vicarious thrill you get just from eyeing the physical tantalizing plates, even if you don’t end up eating one. Compare that to “here’s an empty text box. Whisper your very specific wishes to the genie and maybe it’ll materialize something in the neighborhood of your demand, probably not exactly what you asked for; and it’ll hide all the cool stuff we do that isn’t directly relevant to the demand we made you work to formulate (and probably can’t fill exactly).”
  - acdha 3y ago
    I think Animats has it right: this doesn’t make sense for almost any restaurant - even a place with a huge menu doesn’t have enough entries that a regular search engine wouldn’t be equivalent quality at orders of magnitude better cost & performance – but it could make sense for a company like Uber Eats or Yelp, and I think especially so if the language parsing could handle really imprecise queries across languages. Based on my dinner earlier, a good example would be if you could take “who has that Ethiopian dish with chicken and bits of bread” and return a place listing only “Doro fitfit”, which would require more effort with a traditional search engine since none of those words are commonly present in the menu and you have to have more than basic Solr-style synonym expansion to expand “Doro” to “Doro wat”, which itself expands to “chicken stew”, and expand “fitfit” to “shredded injera”, which expands to “a type of bread”. Obviously that _could_ be solved with the traditional stack but that starts to get more involved so I could see an advantage for making it smoother.
avidphantasm 3y ago
There are already too many restaurants owned by small business tyrants who feel entitled to run a successful small business. If you don't want to make your restaurant menu by yourself, maybe you shouldn't be in the restaurant business?
- steveBK123 3y ago
  Precisely
cadence- 3y ago
Is there any specific reason for using LLAMA embedding models instead of the new models OpenAI just released? Are they better?
- boredemployee 3y ago
  Not an expert but I'd say prolly llama is preferred for privacy + open source reasons.
  - cadence- 3y ago
    Not sure how much privacy is needed for a menu in a restaurant. Plus, it's all later sent to GPT which breaks privacy anyway.
- mvdtnz 3y ago
  There's no reason for any of this bullshit. Like cryptocurrency in 2021 this is all a solution in search of a problem.
  - rwhyan 3y ago
    With this particular solution, I agree. But LLMs and RAG in general couldn't be further from crypto in terms of utility.
cadence- 3y ago
RAG is probably the only thing you can do with LLMs nowadays that is useful, accurate, and cheap enough to deploy in a real business scenario. Every other use of LLMs so far is either too expensive to scale or produces too many hallucinations.
minimaxir 3y ago
The rather complicated prompt to force the recipes to be in JSON w/ constraints can be substantially simplified using ChatGPT's function calling/structured data capabilities: https://news.ycombinator.com/item?id=38782678 https://news.ycombinator.com/item?id=38782678 Additionally, a full vector store is overkill for a menu with <100 items. You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.
- nextworddev 3y ago
  How else would they sell managed vector dbs and llm app frameworks? /s
- all2 3y ago
  > You can just keep a numpy array in memory with vectors for that and use a matmul for RAG. ... what? Tell me more.
  - JimmyAustin 3y ago
    The model they use (BAAI/bge-small-en-v1.5) produces embeddings 384 wide, at float32 that equals 1536 bytes each. The size of all the vectors for 100 items is 153kb. Calculating the dot product of that against a query will be under measured in nano seconds, even with a naive implementation.
    - cheesyFishes 3y ago
      This is actually what llama-index does if you don't use a vectordb integration
m3kw9 3y ago
Nice idea
steveBK123 3y ago
Using a chatbot to answer customer questions about menu items seems like a great way to kill someone with food allergies due to LLM hallucination, but hey.. AI!
- deleted 3y ago
  [deleted]
charcircuit 3y ago
This is way over complicated for querying a single menu. This article seems like a solution in search of a problem. You don't need a search engine when a menu is only going to have a handful of dishes. Just use an picture of the menu as part of the prompt and you will make everyone's lives easier. I'm skeptical that asking questions about the menu has much real utility.
solarized 3y ago
If we've fixed the 24/7 'free' internet with zero latency issue, 24/7 electricity, and cheap mobile devices – in short, ZERO FRICTION – then this kind of tech could all work. Otherwise, we're just adding new problems instead of fixing them.
morbicer 3y ago
Different stokes for different folks but this is dystopian to me. I prefer the approach of standard french restaurant: 2-3 options for each category of appetizer, main course and desert. Don't paralyze me with choices. Too many options also mean the establishment has no focus - they won't be all good so it's a lottery. If I have question I will ask the waiter. And I am as asocial and introverted as any common nerd.

61 points by byyoung3 3y ago

For a single menu, this is silly. As a search engine for Doordash or Uber Eats, though, it might lead to this guy being acquired.

byyoung3 3y ago
1m obo
dkarras 3y ago
I mean even as a single menu, I'm not a big fan of sifting through the menu of a restaurant, especially if I'm eating because I have to. If I could type (or even better speak) what I'm into and what I'm willing to spend and if there pops up the options, it would be a quality of life improvement for me.

jumploops 3y ago

Semantic search with vector embeddings is great... as long as you maintain an inverted index for traditional full-text search. As a user I sometimes want a "buffalo chicken quesadilla" not disparate buffalo wings and quesadilla recommendations. I believe Elastic has a mechanism that supports both? Haven't tested it yet.

CoastalCoder 3y ago
> As a user I sometimes want a "buffalo chicken quesadilla" not disparate buffalo wings and quesadilla recommendations. Curse you! I just started eating dinner, but now I want what you listed instead.
- all2 3y ago
  Uhh, let's see here Roasted buffalo chicken thighs, smoked gouda, and sautéed onions, bell pepper (and jalapeño or other peor is optional, there's a smoked canned pepper -- I cannot remember the name -- that could be especially good here). This is pretty straight forward: 1. Place chicken thighs (bone in, skin on) on a roasting sheet, liberally salt. Roast in an oven at 400F for 30 to 40 minutes (start checking at the 30 minute mark). 2. While the chicken roasts, dice an onion, a red bell pepper, and a green bell pepper and sautee in oil of your choice (bonus points for using the drippings from the chicken thighs). Don't cook them all the way through. 3. Cook down canned black beans with a bay leaf, 1/4c of fat of your choice, salt (to taste), cumin (to taste). They're done about the time this concoction resembles a spreadable paste, about 10 minutes over medium heat. 4. Shred the thighs in a bowl, consume or set aside the skin. Add your buffalo sauce if choice. Bonus points if you made your own Buffalo sauce. 5. Heat a cast iron skillet over medium high heat, or a George foreman style press. I'm assuming these are non stick, add enough oil that there is a fine sheen. We don't want to fry the quesadillas (maybe next recipe). 6. Assemble your quesadillas directly in the pan, in this order: cheese (shredded or slice thin is fine), veg, meat, beans, cheese other tortilla. The last layer of cheese is important and will help glue the thing shut. If you're using a skillet, put a lid on it to tap heat and encourage the top cheese to start melting. 7. When the cheese on the bottom has melted and the tortilla is starting to brown, we flip. Get a plate and slide the quesadilla onto it. Then take the skillet and put it on top and then rotate both. If you're using the press, get a second plate and flip. 8. Wait for the new bottom side to brown and melt. Finish with cilantro, lime juice, and something approaching queso fresco. Greek yogurt, sour cream, goat cheese, anything like that will do. Enjoy! now I also want Buffalo chicken quesadillas [edit] Chipotle! It only took an hour to remember.

sebastiennight 3y ago

Aha interesting. I would guess it is missing a bit of nuance, eg a real word query would not be "appetizers with chicken" but something closer to "I'm waiting for my friends, not too hungry, what do you have that's spicy but not too filling? Oh, and I don't eat red meat". This would be much harder with basic vector database+RAG. You would at least need HyDE (hypothetical document embeddings) to act like a real waiter. I find your project very interesting though, and the walkthrough is nice!

byyoung3 3y ago
Agreed. Rag isn't ideal for every scenario. Putting the contents of the menu in the prompt could even be feasible with a smaller model.
dartos 3y ago
I feel like HyDE would just cause hallucinations. Imagine the generated documents based on that query was like “Our shepard’s pie is the perfect spicy meal for sharing with friends without being filling” (Obviously this is wrong, but it’s an example of a statistically possible document) Then the following similarly search would also be searching for shepherds pie. I don’t think throwing more LLMs at rag problems is the answer.

chadash 3y ago

> Physical menus are fine. They do the job. But if we're looking to actually improve upon them, current approaches miss the mark. They offer limited interactivity and flexibility. Sigh. Am I the only one who likes to hold a physical menu and browse it? I never go into a restaurant knowing what I’m gonna get… I like to see how the menu is themed and decide from there. The UI on a physical menu is a vast improvement over any digital menu I’ve seen.

flextheruler 3y ago
Yeah I’m not sure which restaurants would use something like this maybe food trucks or casual places that already do the QR pdf menus. The menu is a huge portion of setting the ambience and theme of a nice restaurant.
- alwa 3y ago
  And beyond the setting and ambiance, it’s a huge portion of the restaurant’s value proposition: “Here are all the things we do and are proud of, what sounds best?” At the risk of being extra curmudgeonly, have you ever dined someplace where they wheel a dessert cart round? There’s kind of a lightweight vicarious thrill you get just from eyeing the physical tantalizing plates, even if you don’t end up eating one. Compare that to “here’s an empty text box. Whisper your very specific wishes to the genie and maybe it’ll materialize something in the neighborhood of your demand, probably not exactly what you asked for; and it’ll hide all the cool stuff we do that isn’t directly relevant to the demand we made you work to formulate (and probably can’t fill exactly).”
- acdha 3y ago
  I think Animats has it right: this doesn’t make sense for almost any restaurant - even a place with a huge menu doesn’t have enough entries that a regular search engine wouldn’t be equivalent quality at orders of magnitude better cost & performance – but it could make sense for a company like Uber Eats or Yelp, and I think especially so if the language parsing could handle really imprecise queries across languages. Based on my dinner earlier, a good example would be if you could take “who has that Ethiopian dish with chicken and bits of bread” and return a place listing only “Doro fitfit”, which would require more effort with a traditional search engine since none of those words are commonly present in the menu and you have to have more than basic Solr-style synonym expansion to expand “Doro” to “Doro wat”, which itself expands to “chicken stew”, and expand “fitfit” to “shredded injera”, which expands to “a type of bread”. Obviously that _could_ be solved with the traditional stack but that starts to get more involved so I could see an advantage for making it smoother.

avidphantasm 3y ago

There are already too many restaurants owned by small business tyrants who feel entitled to run a successful small business. If you don't want to make your restaurant menu by yourself, maybe you shouldn't be in the restaurant business?

steveBK123 3y ago
Precisely

cadence- 3y ago

Is there any specific reason for using LLAMA embedding models instead of the new models OpenAI just released? Are they better?

boredemployee 3y ago
Not an expert but I'd say prolly llama is preferred for privacy + open source reasons.
- cadence- 3y ago
  Not sure how much privacy is needed for a menu in a restaurant. Plus, it's all later sent to GPT which breaks privacy anyway.
mvdtnz 3y ago
There's no reason for any of this bullshit. Like cryptocurrency in 2021 this is all a solution in search of a problem.
- rwhyan 3y ago
  With this particular solution, I agree. But LLMs and RAG in general couldn't be further from crypto in terms of utility.

cadence- 3y ago

RAG is probably the only thing you can do with LLMs nowadays that is useful, accurate, and cheap enough to deploy in a real business scenario. Every other use of LLMs so far is either too expensive to scale or produces too many hallucinations.

minimaxir 3y ago

The rather complicated prompt to force the recipes to be in JSON w/ constraints can be substantially simplified using ChatGPT's function calling/structured data capabilities: https://news.ycombinator.com/item?id=38782678 https://news.ycombinator.com/item?id=38782678 Additionally, a full vector store is overkill for a menu with <100 items. You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.

nextworddev 3y ago
How else would they sell managed vector dbs and llm app frameworks? /s
all2 3y ago
> You can just keep a numpy array in memory with vectors for that and use a matmul for RAG. ... what? Tell me more.
- JimmyAustin 3y ago
  The model they use (BAAI/bge-small-en-v1.5) produces embeddings 384 wide, at float32 that equals 1536 bytes each. The size of all the vectors for 100 items is 153kb. Calculating the dot product of that against a query will be under measured in nano seconds, even with a naive implementation.
  - cheesyFishes 3y ago
    This is actually what llama-index does if you don't use a vectordb integration

m3kw9 3y ago

Nice idea

steveBK123 3y ago

Using a chatbot to answer customer questions about menu items seems like a great way to kill someone with food allergies due to LLM hallucination, but hey.. AI!

deleted 3y ago
[deleted]

charcircuit 3y ago

This is way over complicated for querying a single menu. This article seems like a solution in search of a problem. You don't need a search engine when a menu is only going to have a handful of dishes. Just use an picture of the menu as part of the prompt and you will make everyone's lives easier. I'm skeptical that asking questions about the menu has much real utility.

solarized 3y ago

If we've fixed the 24/7 'free' internet with zero latency issue, 24/7 electricity, and cheap mobile devices – in short, ZERO FRICTION – then this kind of tech could all work. Otherwise, we're just adding new problems instead of fixing them.

morbicer 3y ago

Different stokes for different folks but this is dystopian to me. I prefer the approach of standard french restaurant: 2-3 options for each category of appetizer, main course and desert. Don't paralyze me with choices. Too many options also mean the establishment has no focus - they won't be all good so it's a lottery. If I have question I will ask the waiter. And I am as asocial and introverted as any common nerd.