12 ms·
This problem is known as "attribution" - you have a "no" or "without" in the sentence, but you don't know where it belongs. One could (and one does) argue that
by DenisM 7y ago
This problem is known as "attribution" - you have a "no" or "without" in the sentence, but you don't know where it belongs. One could (and one does) argue that the problem cannot be solved with statistical methods (ML), especially not in any domain where accuracy is required, such as medical recored analysis: "no evidence of cancer" and "evidence of no cancer" are very different things.
Zooming out, the language field breaks into several subfields:
- A large group of Chomsky followers in academia are all about logical rules but very little in the way of algorithmic applicability, or even interest in such.
- A large and well-funded group of ML practitioners, with a lot of algorithmic applicability, but arguably very shallow model of the language fails in cases like attribution. Neural networks might yet show improvement, but apparently didn't in this case.
- A small and poorly funded group of "comp ling", attempting to create formalisms (e.g. HPSG) that are still machine-verifiable, and even generative. My girlfriends is doing PhD in this area, in particular dealing with modeling WH questions, so I get some glimpse into it; it's a pity the field is not seeing more interest (and funding).
- animalCrax0rz 7y agoThere is this, too, from 4 years ago, which seems reasonable in my not-very-well-informed opinion (speaking of which, I'm not sure the work referenced here can deal with hyphenated negation, but it should be simple to include) https://www.aclweb.org/anthology/P14-1007.pdf https://www.aclweb.org/anthology/P14-1007.pdf code: https://github.com/ffancellu/NegNN https://github.com/ffancellu/NegNN
- DenisM 7y agoThank you, I knew my effort was not for naught!
- smoyer 7y agoWhile this is indeed an example of the attribution problem, I'd argue that this particular query will never be solved. I don't search for a "shirt without stripes", I search for a "solid <insert color here> shirt, or a "<insert color here> hawai'ian shirt". I'd be curious to see how many sentences with attribution problems actually have other structural issues. If I want to write clearly and without ambiguity, I rewrite sentences that have these problems. Why wouldn't I do the same for search queries?
- derivativethrow 7y agoConsider the query, "non-glass skyscrapers", which suffers from the same problem. What do you call a skyscraper like that if you want to refer to it? They exist, but you can't find them using that search term on Google.
- js2 7y agoI'd call it windowless: https://www.google.com/search?q=windowless+skyscraper&tbm=isch https://www.google.com/search?q=windowless+skyscraper&tbm=is...
- DenisM 7y agoYou missed the Seattle Tower Building. It has windows, but very little in the way of visible glass. https://www.emporis.com/buildings/119453/seattle-tower-seattle-wa-usa https://www.emporis.com/buildings/119453/seattle-tower-seatt... Windowless is a superset of glassless.
- bscphil 7y agoTo be clear (for the benefit of anyone else who reads this), the windows in the Seattle Tower are made of glass, but the exterior of the building is not that modern all-glass-and-steel look[1]. This is a third interpretation of non-glass I hadn't thought of and it took me a minute to figure it out. "non-glass skyscraper": 1. No glass used in the exterior construction at all -> implying no windows 2. No glass used in the exterior construction at all -> implying the windows are made out of something other than glass 3. A skyscraper in which glass is not a prominent architectural feature, but the building does contain features like windows and doors that contain glass. (This comment) [1] https://en.wikipedia.org/wiki/Wilshire_Grand_Center https://en.wikipedia.org/wiki/Wilshire_Grand_Center
- hrktb 7y agoIt comes down to what is considered "windows". When the whole external material is glass, glass panes aren't just windows anymore. That's the full glass buildings returned in your windowless query.
- onurcel 7y agoThe point of the OP is that they claim they understand everything. Example: https://www.blog.google/products/search/search-language-understanding-bert/ https://www.blog.google/products/search/search-language-unde...
- mherdeg 7y agoOK, but what if an Amazon algorithm has actually learned that people who search for "shirt without stripes" are more likely to buy more things if the first image they see is a picture of a striped shirt?
- DenisM 7y agoIt's a tricky balance between sales and relevance. If you don't watch for relevance you will end up showing only booze and porn in the commerce search results, border-line-porn in video search (true story), and so on. edit: fixed typo born -> porn
- viridian 7y agoCould you explain the border-line-born bit? I don't get it.
- giggles_giggles 7y agoIt's a typo, he meant porn
- AmazingTurtle 7y agoOK, what if not? See, your comment makes no sense
- cscurmudgeon 7y agoIsn't this just a version of "God works in mysterious ways" applied to AI? "The AI works in mysterious ways. Trust it."
- KorematsuFred 7y agoI am little surprised with this result. When I worked on similar products we constantly look at our query stream, sorted the high volume queries and manually intervened to present better results to our users. I will not be surprised millions of dollars are being lost because of this substandard query result per year.
- DenisM 7y agoI think these hyper-companies are really attached to doing things "at scale" and sometimes, or even often times, they get carried away with that.
- aero142 7y agoIf the problem is that it didn't know where to apply the without, then why does it show me results from only a single entity? I would prefer so see an interleaved set of results containing all ambiguous entities.
- DenisM 7y agoI think that given the complexity of the problem they don't even try to parse the sentence and do attribution - they just shotgun with ML and hope for the best.
- Aardwolf 7y ago> but you don't know where it belongs Yes, the English grammatical rules make it unambiguous where it belongs. This is solvable.
- nailer 7y agoA shirt is an item of clothing. Items of clothing are made from material. Some materials have patterns. One type of pattern is stripes. Seems like a matter for logical inference. At which point it becomes fairly easy to find shirts made from material where that materials pattern is not stripes. But yes, no AI I have seen works reliably on even basic queries like this.
- tsimionescu 7y agoWhat you've described is not simple logical inference, it is logical inference from common sense knowledge. This is an extremely hard problem, much harder than solving attribution in such simple cases. Most likely, common sense reasoning will be required to get full natural language processing, since human communication relies extremely often on such reasoning. But building a knowledge base of common sense facts will be one of the hardest challenges ever attempted in machine learning/artificial intelligence.
- krick 7y agoThis is pointless overcomplicating. I might agree if the example would be slightly more interesting, but "without stripes" isn't even "absence of <stripes>", it is essentially a colour/pattern and can be easily attributed to a range of things exactly the same way "green" can be. Google translate correctly associates much more dubious and abstract concepts than that, and does it with statistical methods, i.e. associating word combinations with a location in a vector space. The fact all major search engines fail to do it here is just shameful. Especially Amazon, where it is pretty much a primary search function.
- tgv 7y agoGoogle translate doesn't understand what it translates either. It just relies on a very large corpus of parallel texts.
- m_sch 7y agocan you formalize/quantify "understand" please
- tgv 7y agoSomething like: build an representation of the text that's language independent and integrated in other knowledge domains, and predict the receiver's reaction to a verbalization of a representation. But it isn't necessary to formalize any of it. At the current level of sophistication, our informal common ground of words like "understanding" suffice for a discussion. It's obvious Google Translate doesn't resemble human language processing.
- joshspankit 7y agoYes, you have points, but they break down here: “Shirt -stripes” is unambiguous to a system, yet the first result on Amazon(.ca) is a striped shirt, and the 3rd is sweatpants.
- pbhjpbhj 7y agoAs someone else said you don't know that's wrong for what Amazon are optimising. If they find people [with your background profile] who buy shirts are susceptible to buying sweatpants, they might also find that if the seed you with "sweatpants" as an idea up front that the repeated presentation of sweatpants in "people who bought X also bought Y" sections is more effective. That's the sort of thing I'd expect Amazon to be doing?
- joshspankit 7y agoTbh, I feel like the underpinning “problem” is that more and more these marketplaces are optimizing for what they want to sell, and seeming to ignore blatant requests. I’m an odd one that I already know specifically what I want to buy before I search for it, but I’m certainly not the only one (and I think everyone has done that at least once).
- deleted 7y ago[deleted]
- joshspankit 7y agoLike, when was the last time this was considered ideal? “Yes, I would like an unstriped dress shirt please” “How about this striped shirt?” “No thank you, I would like an unstriped dress shirt please” “I have some lovely jogging pants” “Ok, I need to be clear here, I would like a dress shirt that has no stripes” “Can I interest you in a white undershirt? People who buy dress shirts usually buy undershirts” ....
- pbhjpbhj 7y agoI think if you were visiting a personal tailor, or perhaps talking to a shop assistant, you might get something akin to that. T: I think pink would look good on you, and it's very fashionable right now. You: Just bring me some yellow shirts to try. T: Oh, I got these, and brought this pink one anyway; try it! But, of course Google isn't making fashion suggestions. But then, ... the tailor might also be just trying to shift excess stock or be on a bonus for selling that particular high-cost shirt.
- ardy42 7y ago> This problem is known as "attribution" - you have a "no" or "without" in the sentence, but you don't know where it belongs. One could (and one does) argue that the problem cannot be solved with statistical methods (ML), especially not in any domain where accuracy is required, such as medical recored analysis: "no evidence of cancer" and "evidence of no cancer" are very different things. Couldn't you just parse the sentence into a dependency tree and look at the relationships to figure that out? CoreNLP got both of your examples right (try it at http://nlp.stanford.edu:8080/corenlp/process http://nlp.stanford.edu:8080/corenlp/process, can't link the result directly).
- njharman 7y agoless than lay person, but in your example > "no evidence of cancer" and "evidence of no cancer" are very different things. Why is it not as simple "no belongs to the word it precedes" ? like unary operator, ! (not), in typical computer languages.
- d1str0 7y agoBecause theres a big difference between english and computer/query languages. You can deduce a lot from a query language, but not necessarily english.
- munk-a 7y agoYou need to actually build up noun comprehension though, because "evidence of no remaining cancer" or other qualifier words can greatly confuse the situation. You'd also have quite a bit of fun trying to parse the phrase "no means no" or other usages where "no" is being used as a noun... And for bonus points, folks talk to search engines in broken english all the time so "shirts no single striped" is a totally reasonable query to submit to a server and expect to be parse-able.
- bad_user 7y agoThe broken English aspect also happens because we know search engines don't understand English anyway.
- munk-a 7y agoI don't know the usage stats on AskJeeves but that search engine specifically purported itself to do well when asked a question in english. Now, the tech at the time wasn't even near close to being able to support it, but their advertising targeted it. That all said I never typed in "Who is Derek Jeter?" and I don't know anyone who did. I think the basic issue is that people just don't respect machines and want to minimize the amount of effort spend on communicating with them - I don't say "Alexa please bring up songs by Death Grips if you don't mind" I shout "Alexa! Play! Death Grips!" and then yell at it when it misunderstands.
- angara 7y agoFascinating, do you have any links to papers about machine-verifiable formalisms?
- DenisM 7y agohttps://en.m.wikipedia.org/wiki/Head-driven_phrase_structure_grammar https://en.m.wikipedia.org/wiki/Head-driven_phrase_structure... That’s the best I can do, sorry.
- glup 7y agoAs a postdoc in computational linguistics, my go-to example for talks is asking Siri not to show me the weather.
- waltpad 7y agoHas any of these talks been published?
- jldugger 7y ago... Siri doesn't show you the weather by default. I know your point is about shallow parsing, but there's a reason it still kinda works.
- glup 7y agoFor "Hey Siri, don't show me the weather!" I get "Here's the forecast for today"
- jldugger 7y agoExactly. But it was already not showing you the weather before you said anything. The query stymies shallow parses NLP assistants do, but isn't one you would actually give in real life.
- saystupidthings 7y agoYour comment shouldn't be downvoted. It is a bad example to use in NLP talks just because of what you said.
- alberth 7y agoThe # 1 google result for “Shirt without stripes” is this very own HN post https://www.google.com/search?q=%22Shirt+without+Stripes%22& https://www.google.com/search?q=%22Shirt+without+Stripes%22&
- animalCrax0rz 7y agoGoogle is in the business of producing quick results to sell ads while keeping the cost low. If anyone does better, they will likely do it using a costlier algorithm, and if there is more margin in reselling tangible products than ads then Amazon is incentivized to use costlier algorithms that are more accurate. But I think the margin in both cases does not justify the use of algorithms that may be more accurate in some small percentage of cases but a lot more costly.
- larl 7y agoanimalCrax0rx is in the business of producing quick pithy posts to earn more karma while keeping understanding low and making claims that are not evidence based... See how that works? That's not really what's going on. Sure, G. is incentivized to include pages quickly, but they are also incentivized to produce them accurately, and as the above poster indicates, this is quite a hard problem to solve generally. A is also incentivized to sell items. In many cases different algorithms will lead to quantifiably different results. The algorithm changes that work better for the measurement set will be kept and those changes which dont will be discarded. And both A and G do that within different constraints.
- animalCrax0rz 7y agoGoogle is aware of this problem in their search approach. It's a business problem, not a technical one. You're saying the same thing in suggesting they base their decisions on some measurement set. If solving the problem adds complexity, which it certainly will, and there is not enough improvement in accuracy for the majority of cases in their measurement set, why bother? You sound like the kind of person that attacks people for their opinion. So weird, dude.
- 7y ago
- conanbatt 7y agoBelieving this is a problem of attribution, I would expect to see results that are shirts without stripes, or not-shirts with stripes. If it just disregards the word without, well, that's pretty bad.
- peterwwillis 7y agoThis doesn't seem like you need fancy linguistics ML to fix. You take 90s-era search engine tech and add a database of attributes about every kind of product there is, and take a guess at whether "without" is a search term, a search result modifier, or a filter for an attribute of a product. When you display the results, simply ask the user if that was right; if it wasn't, ask them if they preferred the other filter method. Use those responses as a corpus to train the algorithm. I mean, context is key, right? You're on Amazon and your first search term is "shirts". Unless their is a band called "shirts without stripes", the user wants shirts. The rest of the query is probably some filter of that product. You know shirts sometimes have stripes. It's not a one-size-fits-all algorithm, but it's simple enough that the user should end up with the results they wanted.
- snarfy 7y agoIt seems like the attribution problem is an English problem. The query doesn't have to be English. https://www.google.com/search?q=shirt+-"stripes" https://www.google.com/search?q=shirt+-"stripes"
- fiddlerwoaroof 7y agoIt does, if you're running a search engine for a general audience. the `-` operator is useful, but it's almost an admission of defeat: what most users want is to be able to describe what they're looking for in reasonable english and get relevant results back. Having `-` for advanced users is useful, but it's not friendly to the majority of users.
- jaimex2 7y agoJust being silly here but the same issue happens if you do the search in Spanish :) https://www.google.cl/search?q=polera+sin+rayas https://www.google.cl/search?q=polera+sin+rayas
- tsimionescu 7y agoYou've solved the problem of accuracy, but you've now introduced a huge recall problem. Your query will never find a page advertising 'the best shirts without stripes', as it contains the word 'stripes'. To be useful, Google must solve natural language problems. You can't solve natural language problems by using formal language in sine bits of the problem, at least not until we have a full Chomsky-style understanding of the whole of human language.
- usrusr 7y agoEnglish is fine, googling for "stripeless shirt" works well enough. The question "shirt without stripes" is begging us to ask is whether we might want to reevaluate projections of imminent AI takeover.
- danielscrubs 7y agoReminds me of the confusion with negatives and languages: -Vill du inte ha glass? [Don't you want ice cream?] -はい [Yes] Does she want ice cream? Answer: No, she doesn't. I added a not, so she's reversing the answer as Japanese people do. The number of times I've been dumbstruck by this is larger than I'd like to admit, and I'm a coder.
- reubenmorais 7y agoEnglish has similar confusion even for plain, non negated questions. Q: "Do you mind if I sit here?" A1: "Not at all!" A2: "Sure!" Both are valid answers and mean the same thing, the person asking is welcome to sit there. This has always amused me.
- Chris2048 7y agoI'm not sure I understand, "mind" means "object to", so they are asking "Do you [object to] me sitting here?"; "Not at all" == "I do Not [object to you sitting here] at all"
- Myrmornis 7y agoDefinitely what you say is the accepted meaning. But wouldn’t you say A1 is more correct than A2? Possibly relevant, there’s also A3: “Sure I do, last time you sat next to me you wouldn’t shut up.”
- lars 7y agoThat's an excellent example to show why statistical NLP outperforms discrete parsing using logical rules. That A1 and A2 means the same is something that is clear to us from the context. This is something a continuous vector space model can capture and a discrete rules based model cannot.
- mikorym 7y agoI don't think this about a lack of interest. There have been some lengthy discussions on HN about vertical search and how Google doesn't always buy up a small company; they litigate.
- zelphirkalt 7y ago> you have a "no" or "without" in the sentence, but you don't know where it belongs Well, one could argue, that it belongs exactly where anyone entering the query put it. Before "stripes". The problem is often, that search engines try to be too clever, while not offering any kind of switch "exactly those words in this order" and that is just a bad user interface.
- Myrmornis 7y agoI think you’re mystifying a lot of people in this thread! Can you add to your explanation why it’s hard for ML to associate the negation with “stripes”? It seems easy: language => English, in English “without” modifies following phrase, not preceding.
- rjurney 7y agoThe latest embeddings/networks like BERT can handle encoding this logic. They take the surrounding words in context when they're encoded.