10 ms·
Applying BERT models to Search
- aasasd 7y ago> In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question. Funny thing, I've seen people mention right here on HN that for DuckDuckGo you need to adjust the style of your queries. This notion puzzles me, probably because I kept the habit of ‘keywordese’ from the olden days. Most of the time, results are about the same for me in Google as they are in DDG and even in Yandex—with the exception that Google is better at grouping related or similar results, and also if there's one or two sources having the search phrase almost-verbatim then they're at the top in Google. Apparently, I already need to learn talking to the site like it's self-aware, to regret ditching it. Now, if some wondertech helps me to home in on the answer to my exact software or programming troubles instead of hundreds of vaguely related SO posts—I could really dig that.
- luckylion 7y agoFunny that you mention Google's grouping and related SO posts ... I still regularly run into that annoyance that SO hasn't figured out how to do canonical URLs and Google then proceeds to put the identical content from SO on two consecutive spots in the results page. SO does not want to fix it (it has been an issue for years and has been pointed out on meta many times), but I'm confused by Google's fail to consider them duplicates when their content is nearly identical (the difference would be just "hot network questions" in the sidebar etc which depend on the time of the page request).
- naringas 7y agoI wish they would let us use the "keywordese" engine as it was 10 years ago if we want to instead of funneling us along with everybody else into the newer intent search engine.
- binarymax 7y agoThis is the kind of search you see everywhere. 95% of queries I see for customers in most fields are only 1 or 2 words long (and they are typically a noun phrase).
- colechristensen 7y agoMaybe they're getting better at natural language, but for me Google searches have been getting gradually worse year after year. I want Google, not AskJeeves. The "frustration" is "increasing" "when" I "have" to "quote" nearly every "word" to get Google to actually return results with what I searched for instead of what it thinks I meant to search for. And there's the frustration, computing today tries as hard as it can to figure out what it thinks I actually meant. I don't know if it is worse that a person who knows what they want can't get it when the computer disagrees or if the computer is actually mostly right and its algorithms start to push your desires in it's own direction and whatever motive. Facebook already does this really, radicalizing people by engineering the most dopamine-driving content to the top either towards self-obsession or an us v. them bubble. In other words, I just want a fucking regular expression instead of our new data-science overloads ruining our minds with artificial non-intelligence (for profit).
- thorwasdfasdf 7y agoYup, this is a huge problem with google search these days. It keeps trying to figure out what your intent was, and although it works about 70% of the time. the other 30% is a complete disaster.
- mxuribe 7y agoYeah, when i used google more often in the past, my experience had been more like 60% (google works) vs 40% (disaster). Now that i use duckduckgo as my primary search engine, the resultant hits haven't really improved, but at least i feel like i have more privacy. ;-)
- oarabbus_ 7y agoEspecially when it comes to "less acceptable" things. Suppose, for instance, one listened to a rap song and then googled "lean and sprite" (codeine cough syrup and soda, commonly referenced by rappers as a way to get high). Instead of returning search results of relevant rap songs, you get a bunch of links on drug abuse, rehabilitation centers, etc. Thanks for assuming I'm a harrowed drug addict, Google, but really I was looking for music.
- sargram01 7y agoSearch and a lot of the AI based systems these days feels like talking to a hard of hearing grandparent; as long as you’re saying about what they expect then it’s fine, but if there’s any nuances or homonyms it turns into a comedy routine.
- Pick-A-Hill2019 7y agoAn interesting snippet from TFA - "..with this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU." https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html https://ai.googleblog.com/2018/11/open-sourcing-bert-state-o...
- deleted 7y ago[deleted]
- amiga-workbench 7y ago>By using new neural networking techniques to better understand the intentions behind queries Great, so the search results are going to get even worse?
- mankyd 7y agoI am curious why you take the pessimistic view. Their own testing shows an improvement for at least 10% of their queries: "Google says it can now offer more relevant results for about one in 10 searches in the U.S. in English"
- luckylion 7y agoThe improvements are likely for non-power users. As they make up the majority of users, it's understandable that Google optimizes for them. Still, it gives power users worse results because they know what they are looking for and they (often) use certain words for a reason. Google's default assumption ("the users have no clue what they are doing") leads them to a paternal "I think I know what you're going to ask ..." approach which often fails and leads to shit results. The method to combat that is to start wrapping every word in quotes which is annoying. I'm sure intermediate users will catch on at some point, start doing it too and Google will drop the quote modifier. Let's keep it a secret so that happens later rather than sooner.
- mankyd 7y agoThis assumes a simplistic view of how its core search algorithm works, though. It's obvious through even some basic querying that it uses different strategies for recognizing intent based on the nature of the input: single words, addresses, arithmetic, business names, full text queries, etc. There's no reason to think that they will enact this to the detriment of other queries. It _could_ happen, but I am skeptical - optimistic even. As they mention, this improves ~10% of queries. The other 90% likely represent different forms of query input and I would hope remain unaffected.
- minwcnt5 7y agoNatural language is a lot more expressive than boolean expressions of keywords. Perhaps experienced searchers who use the same approach to query formulation that worked well two decades ago are no longer the true power users. Maybe they are just dinosaurs.
- vuln 7y agoNow google is trying to read my mind, my thoughts and my intentions. I've been running away from EVERYTHING Google.
- packetslave 7y agoAnd yet... $ host -t mx vuln.ninja vuln.ninja mail is handled by 10 alt3.aspmx.l.google.com. vuln.ninja mail is handled by 1 aspmx.l.google.com. vuln.ninja mail is handled by 5 alt1.aspmx.l.google.com. vuln.ninja mail is handled by 5 alt2.aspmx.l.google.com. vuln.ninja mail is handled by 10 alt4.aspmx.l.google.com. Maybe you should try running a bit faster.
- Pick-A-Hill2019 7y agoOuch. The burn.
- cromwellian 7y agoThere's a lot of virtue signaling on HN.
- bratao 7y agoBERT is truly amazing. Almost all inovation in NLP uses BERT and transformers somehow. ALBERT will be the next HUGE thing for the next months, as it show results better than BERT with a small fraction of parameters. We did a "Semantic Similarity search" for some documents, where we represent a document as a vector using BERT, and had to look for documents close to a reference document. The results where breathtaking. It really returned semantically similar documents. You can do it now using ElasticSearch(But you really should do it using Vespa.ai, it is much faster https://github.com/jobergum/dense-vector-ranking-performance https://github.com/jobergum/dense-vector-ranking-performance )
- buboard 7y agobut the question is how better they are compared to existing similarity measures. E.g. for documents in a domain, even simple cosine is pretty good.
- bratao 7y agoI did not understood. BERT it is not a similarity measure. For our use case we did use a simple cosine similarity to find the similar documents. But we have to represent those documents in a vector space. From our test representing the document as a MEAN of BERT Embeddings we got some very good results. Much better than BoW, Glove or the Lucene "More like this"
- codingslave 7y agoI wish google would release google2008.com On that website, they keep using their technology from 2008 and let me use it to search for what I want. I've had enough
- windsurfer 7y agoOne issue with that is that there has been an arms race since 2008 with "search engine optimization". If you used a crawler from 2008 on today's web, you'd probably get a ton of spam.
- codingslave 7y agoThe results for half of my searches are already all spam. It would be better to just accept that and have google let me engineer my queries so that I avoid spam. More token based searches, less word vector/machine learning based search. Let me query their index like its an SQL database
- windsurfer 7y agoAre you sure they are spam? Google's engineers consider annoying things like recipe pages with 20 paragraphs of stories before the actual recipe not to be spam.
- buboard 7y agoi am being tricked into pinterest link-pages multiple times a day. That is spam ,and has very little to do with their AI work
- im3w1l 7y agoRestricting it to 2008's web could be an interesting exercise.
- drusepth 7y agoI'm now super curious what percentage of pages from 2008 are still around/available today.
- binarymax 7y agoI've been working on using BERT for search, for research and training development, with not so great results. Note the quote "when it comes to ranking results, BERT will help Search better understand one in 10 searches". This is because of the "keywordese" point they noted earlier in the article. Most searches are 1 or 2 words - there isn't enough to grab onto for meaningful ranking with short queries and a similarity function for longer text documents. Also, try keeping the systems afloat to handle search like this. BERT is not practical to use for search results by anyone without the scale of a company like Google. You need to have a server farm of GPUs to translate all your documents into tensors - and then keep them around somehow! A document of 10k text will balloon to ~1MB when converted to a multitoken vector representation. BERT uncased has 768 features - thats 768 floats per token you need to keep around. If you compress it using PCA or averaging across tokens, you lose all the juicy context that you need for the matching and ranking. Also, there currently isnt a good way to keep this stuff around yet (though there are active projects ongoing to get this into Lucene [1],[2]) I think this is definitely a great achievement in NLP - but it needs breakthroughs in other areas to be useable by product teams implementing search, with any reasonably large content size. [1] https://arxiv.org/abs/1910.10208 https://arxiv.org/abs/1910.10208 & https://github.com/castorini/anserini/blob/master/docs/approximate-nearestneighbor.md https://github.com/castorini/anserini/blob/master/docs/appro... [2] https://github.com/o19s/hangry https://github.com/o19s/hangry
- pheug 7y agoDistillation is usually used today to tame its resource problems at scale - you run BERT to squeeze out maximum signal from your training data and then distill the model e.g. into cheap CNN for inference.
- binarymax 7y agoDistillation reduces accuracy and removes the contextual precision. For example reducing a whole document to some N (1k or so) dimensions have worked very poorly in my experiments for short queries - typically making the relevance worse than basic keyword search.
- 6gvONxR4sf7o 7y agoI wonder how this affects their costs of serving a query. BERT isn't exactly computationally cheap to evaluate.
- tanilama 7y agoIt is google, it is probably their secret sauce. I won't be surprised that someday Google makes custom ASIC chip to just run transformer models.
- 6gvONxR4sf7o 7y agoThat's actually old news, and mentioned in OP. They build their own chips (TPUs) that are super cool. You can even use them as part of google cloud! Still, moving to BERT ain't cheap.
- tanilama 7y agoTPU is old news. I think what would the actual news will be a Chip that is customized/optimized to just run BERT. The Transformer architecture itself stays mostly unchanged in the 2 years after it has been proposed, and with BERT/variants, most (competitive) NLP models are now Transformer based, it makes sense to make custom chips to just run Transformers, the same as CNNs.
- sdenton4 7y agoMeh, not sure how much more there is to do to specialize for transformer specifically. TPU and GPU are mainly just fantastic matrix multipliers. And transformer is partly designed with this hardware in mind: the operations are basically the same operations you see in a CNN. And in fact, one of the nice parts of the transformer is that you can run it without RNN, making it even better optimized for the matrix multipliers. Furthermore, tpus are a moving target themselves: as ML needs change, the team build new operations and optimizations into the next generation of chips.
- summerlight 7y ago
- braindead_in 7y agoHow long till NLG becomes good enough that it can answer questions factually? I think integrating it with a knowledge graph might just make search obsolete.
- ur-whale 7y agoI might be mistaken, but I believe that's already what Google is doing for a lot of factual knowledge (as a matter of fact they do call it knowledge graph). Try for example to type "when was jfk born" in Google, you should see a factual answer fished from a kg.
- deepnotderp 7y agoBERT is such a great case of clever hans. If you scrub the shallow statistical similarities from question answer sets accuracy drops significantly .
- lexpar 7y agoDoes anyone have a nice resource they recommend on what BERT does? I've gathered it was trained by trying to predict missing words in a sentence, but I don't have an intuition on how this is useful for downstream prediction (like, say, learning a word embedding is).
- ma2rten 7y agoThis blogpost is very light on details. It doesn’t at all say at which stage in the search process BERT is used.
- vagab0nd 7y agoTotally off-topic. Is it just me, or is this "People also search for" the worst feature ever? For those who haven't noticed, this is the box that shows up under a search result when you return to google from the page you've gone to. 50% of the time I click the back button, this freaking box shows up, the whole page shifts, and I click the wrong link. Oh and did I mention I never ever, not even once, actually use it?
- breadandcrumbel 7y agos there any public information on actually how BERT is being applied to IR? For each of the scenarios they described they are just like "here's potential hard search query, and BERT adds magic language understanding which makes it all better ". It's non-obvious how BERT is actually being used though, especially at the scale and latency they need. (I get that that this is Google's "secret sauce" and they might not saying anything in this particular use of BERT. But I'm curious if anyone had seen anything related.)