11 ms·
Google's AI doesn't understand restaurant menus
- Sreyanth 4y agoI see a CAPTCHA opportunity here. Show the line item and ask people to type both the dish name and price in two boxes. Or may be give a couple of options that the end user must confirm before they can proceed.
- hunter2_ 4y agoYep, anyone remember having to key in house numbers from street view photos? Similar idea. But I think human/bot differentiation no longer demands so much effort of users lately, so this labor pool could be a relic of the past.
- mellavora 4y agoHoy, boy, is that going to be fun! Imagine having to properly label thai food items to view websites! Or register for government services!
- tgv 4y agoThey'll just train an even bigger network.
- jgalt212 4y agoThey can't because grokking restaurant menus is a moving target. Restaurants are constantly looking for new ways to describe new things and new ways to describe slightly updated things.
- ethbr0 4y agoI've found adjective count on the menu to be inversely correlated with food value (quality/price). And am not joking: it's pretty damn reliable. E.g. "chicken" vs "locally-raised, hormone-free poultry"
- briandear 4y agoWhy can’t a restaurant simply have an RSS feed for the menu and google use that? Very easy to do. Can simply use the menu data from the POS system.
- MontyCarloHall 4y agoI’ve found that these sorts of annotations are a lot better in the US than in other industrialized countries. In foreign countries, points of interest in Google Maps are often missing basic info like opening hours or even telephone numbers. Often times the actual location is misplaced on the map. When it comes specifically to restaurant menus in the US, most seem to be manually transcribed by the restaurant staff. The items and prices are correct (but often out of date), and food descriptions faithfully reproduce non-native spelling/grammar mistakes. In addition, I almost always see user-uploaded photos of the menus. This does not point to a difference in Google’s automatic parsers or in the level of Google-generated content; it seems that US users contribute to map and PoI content a lot more. I wonder why this is. My guess is that there are far fewer staff members at Google curating crowdsourced content outside of the US, which makes non-American users much less likely to contribute, since their contributions will appear much more slowly, if at all. I’ve contributed my own corrections to PoI data in the US (e.g. opening hours updates) and seen it reflected on the map in a few days. This probably wouldn’t happen elsewhere in the world.
- ramchip 4y agoIn Japan Tabelog is a lot more popular than Google Maps for restaurant reviews, photos, etc. I think it caters better to the local market.
- yeputons 4y ago> points of interest in Google Maps are often missing basic info like opening hours or even telephone numbers. To be fair, sometimes neither of these exist. A place in some parts of the EU may be open whenever the owner feels like it. You can't even approximate opening hours and holidays unless you actually ask the person over the counter when they're actually open.
- elijaht 4y agoTangential but Google will call restaurants with their automated assistant to verify hours
- 4y ago
- progfix 4y agoDoes Google ask the restaurants for permission to do that? Couldn't they get sued for damage of reputation when they show wrong prices?
- hackmiester 4y agoI’d hope so, but who’s got that kind of money?
- mrweasel 4y agoIt is also a potential copy-right violation, not the I think Google care.
- nmstoker 4y agoNo, generally it won't be, since you cannot copyright facts.
- mrweasel 4y agoWhat about titles on signature dishes?
- nmstoker 4y agoNo again :) https://copyrightservice.co.uk/services/knowledge-base/kb_name#:~:text=Copyright%20does%20not%20protect%20individual%20names%2C%20titles%20or%20phrases https://copyrightservice.co.uk/services/knowledge-base/kb_na....
- nmstoker 4y agoThey could possibly register a trademark, but few restaurants seem to. I can't imagine this creating much of a barrier - Google would stop if told to by the restaurant but then they might not list the restaurant, and what restaurant would see that as worth it?
- mattzito 4y agoFwiw, this at least partially not google’s fault. Just about everybody uses one of a few companies for restaurant menu data, and the biggest one is far and away SinglePlatform. SinglePlatform is still powered largely by scanning menus and having contractors enter them by hand, which is where many of these errors happen. More forward-looking restaurants manage this all themselves as part of their digital strategy, but it’s still a small percentage and disproportionately located in the US. I’m not saying Google doesn’t also scrape or use other sources, I’m sure they do, but this is one of those situations where the whole system is broken. Tbh one of the bank shot benefits of having all of these digital delivery services is that some restaurants are using aggregators that can also publish menu data. As far as the authors idea about markup for menus, that’s great, but highly improbable for a bunch of reasons: most restaurants don’t update their menu frequently, dishes are often difficult to represent structurally, POS systems are often modeled differently than the printed menus, etc.
- tialaramex 4y agoThe most notable restaurant near me, (Garden Restaurant) can't even spell its own name correctly on its expensive professionally produced menus in the restaurant, your chance of scraping good enough menu data from their web site is negligible. I actually went to the web site just to see, and it's worse than I thought. Even their Western menu, the stuff random Westerners think is "Chinese food" is presented as JPEGs of photographs (sometimes out of focus) of the physical menu, which is itself strewn with typographical errors and mysterious annotations. So, to get even the bad text an actual patron has in the physical restaurant you need to scrape the site, download the images, and successfully OCR from low resolution out-of-focus photographs. It's not impossible but good luck to you, and at the end the results will still be pretty unsatisfactory. "Frind pok" is actually what they wrote, they meant Fried Pork, but that's not an OCR error it's really what they paid to have printed.
- passivate 4y agoSince its near you, have you tried going there and explaining it to the manager?
- logicalmonster 4y agoRegarding the price discrepancies, why can't they program some kind of outlier check into this type of system? If an average burger costs $12.00 in an area, but the system interprets the price as either $1,200.00 or $0.12, it should be relatively easy to flag that as a likely error.
- edent 4y agoThat doesn't come up on LeetCode so must be pretty difficult to implement.
- ChuckNorris89 4y agoI'm not sure if you're being satirical or not but I definitely chuckled.
- hoosieree 4y agoIf it's not on LeetCode it's not a real problem.
- kwertyoowiyop 4y agoNew problem: invert this menu and sort it by the third letter converted to lower-case, filtering out seafood. Present all item descriptions with infix notation.
- ohgodplsno 4y agoWhat, do you mean reversing an octree in the 4th dimension doesn't solve that ? But it's the only thing I practiced for my interview!
- isoprophlex 4y agoI'm sorry, no, you must be mistaken about the sheer scale at which google operates. Sanity checks simply don't scale, you see. By necessity, it'll be "pour it into this big stew of linear algebra" or nothing. Any kind of sensibility is simply impossible when you're as big as google!
- ethbr0 4y ago
- faddypaddy34 4y agoThe biggest problem with Google and restaurant menus IMHO is the fact that the actual restaurant website (if the have one) is often several links below or often off screen to a bunch of review sites and other sites like Google maps with unreliable information.
- jthrowsitaway 4y agoGoogle likes to push you into a textual/parsed format of the menu, which is often incomplete and difficult to navigate. Just show me the pictures of the menu, please.
- dawnerd 4y agoAnd make it obvious when the menu is from. Too often menus are years old. And with prices going up overnight everywhere pricing has gotten really hard to figure out.
- amelius 4y agoGoogle's AI doesn't understand much. For example, a Google search for "Jeans $BRAND waist $WAIST length $LENGTH" already is too much to ask.
- visarga 4y agoIt seems to be unable to parse word order, like a "bag of words" model from 2000. For example, if I search "similar pairs of sentences", it replies with "similarity between two sentences". No, I want a pair of sentences (A1, B1) that is similar to another pair (A2, B2). I am not talking about similarity between sentences, but between pairs. The distinction is lost on Google. And they claim to be using transformer neural nets for search. Pfft!
- yunohn 4y agoThis is actually very true. Any such query that is specific gets normalised into something uselessly general.
- enos_feedler 4y agoI don’t trust Google’s extraction of anything. I follow links and read myself. I got burned too many times for open hours especially. Example: https://www.google.com/search?q=stanford+dish+hours&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=stanford+dish+hours&ie=UTF-8...
- weird-eye-issue 4y agoThe hours appear correct for me On the site it says "April-August Public Access Hours: 6:00am-7:30pm" And I see the same on Google
- enos_feedler 4y agoSo actually, it’s either fixed or its a broken clock is right twice a day. I google regularly to get to the page at the beginning of each month and it doesn’t extract based on time of year. It just picks one. But maybe it’s been fixed! :D
- weird-eye-issue 4y agoI don't think the hours are pulled from the webpage. You can even go and suggest changes to the hours yourself on Google
- thomasahle 4y ago6am to 7.30pm. That seems correct from the website for June. Though I guess it may get confused some months, or lag behind?
- enos_feedler 4y agoI worked at Google helping news publishers add metadata to extract live/evolving news coverage for the real time / breaks coverage carousel above the search results. Google will never believe what authors use for meta data. It is just a hint and that will always be the case. There is too much opportunity for deception.
- edent 4y agoI can understand that problem for some sectors. But what's the advantage to a restaurant in publishing an inaccurate menu (fake foods!)?
- vander_elst 4y agoMaybe fake prices... It could help lure customers in.
- enos_feedler 4y agoI don’t know, but maybe the metadata prices are inaccurate so that the query “burgers under $8” surfaces that menu. There are just all kinds of reasons to game the system to get your results in the page. Trusting meta data over user visible information just opens a portal to these kind of things where you tell the search engine one thing for SEO and show the user something else. You could police this of course, but it’s much easier to just extract the info from the page since there is more incentive to be accurate for readers.
- madisp 4y agoperhaps instead of nefarious usecases it could just be out of date
- enos_feedler 4y agoYes this too
- wzdd 4y agoI'm not really convinced by the "intentionally inaccurate" arguments (if they want to deceive, then surely they could also just serve a fake PDF to Googlebot). But I suppose it's reasonable to assume that restaurants would be less likely to keep their metadata up-to-date than they would their PDF menu. Unintentional inaccuracy, in other words.
- ape4 4y agoGoogle Maps already asks regular users questions. It could ask if the menu is accurate.
- jeffbee 4y agoSo this guy is just angry that Tesseract isn't perfect? I hardly see the point of this post.
- asveikau 4y agoTo me it points out that they could put additional filters (something to flag entries for the £2000 meal, or USD being used in the UK), or greater manual review, maybe when an algorithm detects such absurdities.
- marsven_422 4y ago
- passivate 4y ago>In the glorious future, every website will be chock-full of semantic metadata. Restaurants won't have a 50MB PDF explaining the chef's vision for organic cuisine That is a good idea, especially for visually impaired users - but why can't we have both? I sometimes like seeing the fancy fonts, and images if I have never eaten that dish before, etc.
- StevenWaterman 4y agoYou can do all that with HTML (with semantic markup) and CSS though
- splonk 4y agoI've never bothered looking at Google's interpretations of menus on Google Maps. When available, I'm just always going to go to the actual restaurant website and look there instead. What I have found occasionally useful is the (presumably automatic) categorization of photos into a menu category. Those at least generally let me know whether something's up to date (especially useful the last couple years as restaurant concepts have fluctuated wildly), and are very helpful for places that don't have much online presence - street food stands, pubs, local fast food places, that kind of thing.
- visarga 4y ago> Google has discovered that it takes 90% of the effort to get 90% of the way there - but the last 10% takes the next 90% of the effort. I can confirm, I am working on information extraction and 90% is the glass ceiling for current models. With much labelled data you can get higher scores but they don't generalise from one vendor to another.
- robswc 4y agoSeems people rediscover every few years that despite how awesome the AI is now, its still just the same statistics under the hood. This is not at all to detract from the hard work and accomplishments made... its just still fairly easy to confuse even the most advanced AI. I guess its impressive that its only easy if you know how they work though, might take the average person awhile to find a way to trip them up.
- chillingeffect 4y agoPrevious generations knew this and left it alone. This generation is happy to provide 90% quality. Neither is wrong, just instances of values
- dash2 4y agoActually, "plain paper" could easily be a "plain poppadum", since the Hindi word is pronounced papaD, with a deep D that is close to an R (https://en.wikipedia.org/wiki/Papadam#Spelling_and_pronunciation https://en.wikipedia.org/wiki/Papadam#Spelling_and_pronuncia...). I've seen this mistake before on restaurant menus, so it may not be Google AI at fault.
- lemursage 4y agoThis reminds me of a story my father used to tell us. Once he travelled to Finland (70s) with my grandfather and they ended up in some restaurant. Having no clue what was in the menu, as neither of them spoke Finnish, they decided to go for the cheapest thing on the menu first. The waiter brought them heated plates. Turns out that the cheapest thing on the menu was a heated plate service.
- idk1 4y agoI did a similar thing, I assumed a cheaper item was the side, and ordered four Dosa as a side in India. Needless to say breakfast was taken care of.
- jahnu 4y agoWhile travelling around China and speaking none of the local languages I resorted to either pointing at other peoples food or picking random items on the menu. I had learned how to ask for rice and beer so that worked well with what was usually a bunch of random but very interesting and tasty dishes. One day while in a border town in the south near Laos, my wife and I were in a suitability weird and humid restaurant with a slow ceiling fan keeping us a bit cool. On the only other occupied table sat a bunch of police with what looked like the local police chief due to the hat on his head but otherwise naked torso. They were just getting drunk so we couldn’t point at food and order. We asked for a menu. I pointed at 6 random things. They gave me a funny look. I then asked for 2 beers in Chinese. They confirmed “two beers?” with an inquisitive look. I confirmed. Then I asked for rice and remembered how to ask for spicy cucumber, a delicious side in china I had come to love. Eyebrows were raised. Shortly thereafter out came two beers, spicy cucumber, and 6 mocktails in tall sundae glasses with umbrellas and curly straws. The police table almost died laughing. Good times :)
- moss2 4y agoThank you for sharing this, I had a good laugh
- abrax3141 4y agoFortunately, misreading a menu won’t drive you into a tree. A friend of mine told me that he can’t use his un-named supposedly intelligent car on the side roads near his home because it reads the house numbers and thinks they’re speed limits. It’s okay going up the long hill to his place when the numbers go down, but going down hill the last number before a down hill sharp curve is house number 65!!! The many joys of our modern AS (as in AI, but not so much) world.
- deleted 4y ago[deleted]
- Barrin92 4y agoI wonder how things would look like if the 'semantic web' (the actual web3?) had taken off and we had regular and rich, machine-readable metadata for just about everything rather than having to rely on what is largely subpar scraping and 'AI' systems. People have pointed out recently how Google search seems to struggle as sites on the internet turn more and more into apps rather than standardized documents and they just go and search on reddit. Having a standard to encode semantics seems honestly necessary at this point if you want to keep things interoperable.
- warkdarrior 4y agoI think Google Search struggles because of SEO/spammed, not webapps. Not sure why SEO will go away in the semantic web.
- warning26 4y agoIf anything, the semantic web would probably have been easier to do SEO spamming with.
- ghaff 4y agoThe semantic web came out of a very academic tradition where scammers and spammers weren't lurking around every corner. Yeah, in addition to its other problems, it wasn't especially attuned with the Internet as it actually came to exist.
- lmm 4y agoGoogle "won" in the first place by being less semantic than their competitors (who used meta keyword tags etc. which made them very vulnerable to spam) and just reading the text of the page (and especially of links to the page) instead.
- colechristensen 4y agoThis is not how I remember Google winning. In those days of search it wasn't spam clogging up search results, it was just plain hard to find things. Google won because you could find things on Google you couldn't find elsewhere. You could probably recreate exactly that phenomenon today, however complicated by the fact that legitimate good-intentioned creators have to behave in much the same way as spammers.
- nikolay 4y agoSentient AI is not interested in food... which it cannot consume.
- bigwheeler 4y agoEven if the restaurant has bothered to create the appropriate machine-readable descriptions, google doesn't bother doing anything with them. Even if the descriptions literally mirror the visual display on the page. I see it all the time, like, on this page (https://www.anthonyspizzabelmar.com/menus/menu https://www.anthonyspizzabelmar.com/menus/menu), which is easily parsed as valid menu schema through the schema.org validator. If restaurants were rewarded with actual updated menus on google, you can bet the restaurants would care about creating the micro data, but it's a waste of time.
- geoduck14 4y agoThe restaurant owners don't need to go through the effort. I'm in a consortium of companies that use ML in their business. One of the companies is a competitor to GrubHub. They use ML to: read scanned menus, understand items, look at pictures... then: predict what ingredients items have; populate if the item is one of entree, meal, desert, or snack; populate if the item is one of gluetn-free, vegetarian, or some other similar stuff All of this without the mom-and-pop restaurant owners lifting a finger. It gives them a competitive edge over their competitors. All of this to say: Google doesn't care - but GrubHub, UberEats, and the ilk do care
- mattcwilson 4y agoWarning: potentially biased opinion. Speaking only for myself, but informed by my job. There are lots of problems with scraping-based approaches. One, yes, you need some really good tech to scrape data from menus, which, even though they are “structured”, next time you’re at a sit-down restaurant, pay attention to all the subtle discrepancies in formatting between different sections/categories on the menu. Two, if the menu isn’t html, but is an image or a pdf upload, now you need some strong OCR on top. Three, the website is generally not likely to be current with what’s actually on offer in the establishment itself. Specials, seasonal dishes, or items that are out of ingredients (“86’d”) will still appear on the menu. That’s going to lead to complaints, refunds, or generally bad customer experience from whoever’s consuming your data / using it to buy food. Four, you’re going to want to to be paid for all this tech and customer support you’re electing to intermediate between the end purchaser and the restaurant, as a service, and so you’re going to tack on some fees and either jack the price up on the consumer or try getting the restaurant to pay you a finder’s fee, cutting in to their already narrow margins. Five, if you’re trying to provide ordering service and not just menu data, you still need to submit the order into the store itself, somehow. Which either means calling it in, robo-submitting an online order (if you’re lucky), or sending a courier to place the order and wait. And then, on the other side, whoever’s taking orders for the restaurant has to punch in the request to the register to actually complete the transaction. Which means the system you really want to talk to isn’t the website, it’s the point-of-sale. Good luck with all that. Source of bias: I work for a company that helps restaurants enable online ordering and POS integration so they can pay much less in fees and focus on making exceptional food.
- s1mon 4y agoI would be happy if Google and Yelp could even get the opening hours of restaurants correct. With Covid, and even now that things have been re-opening and getting busier, so many places have incorrect hours or are outright closed for good, despite active listings. I've defaulted to at least trying to call first if I'm set on going to a particular place, because inevitably if I don't, the restaurant isn't open or is about to close. Of course then the issue is half the time no one answers the phone anymore at restaurants and they don't bother with voice mail. If they do have voice mail, it's still probably got an announcement about the special meal that they offered for Valentines day or Christmas. Sometimes the only solution is just to go see if they are open. To some degree the real issue is that each restaurant can change hours (or menus) at a moments notice, and at many places, the staff and management is not super computer savvy. So no one thinks to update these sources of info, and/or they don't know how. Google has the added data (from tracking phones) of how busy the restaurant is at a given time, but that is presumably some sort of moving average over time, and not necessarily current or accurate. When I first saw the headline for this post, I thought it was going to be about a related issue: even if AI is really good at understanding general spoken/written languages, the names and wording of menus is its own weird thing. If you're then trying to auto-translate that to another language it can be next to impossible. Ethnic restaurants in different places which supposedly speak the same language can have all kinds of spellings and ways of describing the same dish. In the US we call a long sandwich a sub, grinder, hero, or hoagie (to name a few), depending on where you live etc. Or the same name can mean wildly different things.
- alistairSH 4y agoDoes anybody actually rely on Google for this sort of info? Hours, menu, etc are so flaky that I always go to the restaurants website for more info.
- H8crilA 4y agoFWIW there is a nice solution to this, schema.org. I do not know if there's a restaurant menu schema yet, likely there is. In the great jungle of the Internet often you end up with both: a) attempts to process/parse the content as the user sees it, as it is in HTML or some other format, which get b) overridden when the site owner starts to publish the schematized data. The above applies to all web walking robots with non trivial capabilities.
- meroes 4y agoThis reminds me of a place that purposefully puts their hours wrong online. In their own words, they are too busy. Further than static difficulties, there may be a purposeful reaction to make it more difficult for the online world to “digitalize” the non-digital. Good luck abstracting a reactive world.
- noobermin 4y agoThat last bit is a bit too real. We deserve better than having a monopolistic force controlling the web but with all the poor products of a rookie shop simply because they're all hand me downs from engineers climbing the internal google ladder.
- aaaronic 4y agoI worked at a place that was trying to do this for thousands of restaurants, only they wanted to configure the POS system (all the items, variations, modifications, combos, etc.) using just the PDF menu. We never got very far with automating it, but part-time high school employees were usually pretty decent at it. Menu layouts are all over the place, so using OCR was trivial, but trying to figure out what went together and what modifications were valid on which items was a complete mess. Shrimp cocktail w/ cheese, anyone?
- bombcar 4y agoThe best part of these automations is you can end up with hilarious orders - like the McDonald’s app will let you order a cheeseburger with no cheese, no bun, no meat, no onion, no pickle, and no sauces. I suppose they should give you an empty wrapper.
- egorfine 4y agoToday I learned that I am in fact Google AI. As I sometimes don't understand those 50MB PDFs either.
- idonov 4y agoIn Hebrew it sometimes shows "Bad service" as a menu item it if sees it in a lot of reviews.
- dredmorbius 4y agoGoogle should loose it on the My Cousin Vinny diner: https://yewtu.be/watch?v=pWC0sKCS5oA https://yewtu.be/watch?v=pWC0sKCS5oA
- whoomp12342 4y agowell good because I dont eather.