4 ms·
[a crude speculative outline] No. Because to a first approximation, it is not possible to make it work all the time in practice. Results returned are best eff
by brudgers 5y ago
[a crude speculative outline]
No. Because to a first approximation, it is not possible to make it work all the time in practice.
Results returned are best effort. Best effort is largely limited by latency.
At the scale of the entire web, parts of the index will be absolutely inaccessible due to partitioning and practically inaccessible because they are sitting on disk (or simply don’t exist because they wouldn’t fit into memory) or are so far away that they can’t reliably be returned within the limits of acceptable latency — e.g. on a server in Japan for a London query (and anyway how likely is relevance of Japanese results?).
I mean maybe there are results for the “site” qualifier in memory on the server your query hits and maybe their aren’t. Either way, Google’s going to return results in 250ms if that’s the performance target. Either way Google’s not going to spin for twenty minutes or seconds it would take to crawl the site, process the results, and build the inverted index.
Everything is engineered. The engineering is guided by statistics. But so is performance. Sometimes better information is close by in hot memory. Sometimes it isn’t. Sometimes bandwidth is saturated. Sometimes your cell service is slow.
That’s probably why HN uses Algolia even though it costs money.
- justinclift 5y ago> No. Because to a first approximation, it is not possible in practice. That doesn't really seem to fit how things work. It "used" to work, at least with DDG until a few months ago. eg, giving a search query of: "some specific IT term" "some other specific term" linux Would actually return pages with those first two terms in them, and a "best effort" for the last term. However, now their results are often garbage. It's most likely the search will have a list of results with neither of the first two "exact" terms in them, and may only have some resemblance to the third term. Not sure what's going on, but it's been a huge step backwards and I'm now personally open to using an alternative search engine that actually works. _Not_ Google nor Bing though.
- brudgers 5y agoSorry for not being clear. I have updated my comment. I meant it is not possible to make it work all the time. In my experience, Google etc. have always done better when the search is about computing, e.g. when it is about IT terms. In the long tail away from computing and a very broad definition of popular culture, it is not uncommon for quotation to make no difference at all. Particularly when the quotation has a slight resemblance to something broadly in popular culture...or god forbid a product. It is of course possible to get exact matches in theory. I mean your favorite text editor can do it in your source files. And in the early 1990's Web Crawler could do it for the web. But today, it's cosine similarity plus special exceptions looking at an arbitrary subset of the entire sharded index that happens to be in hot memory of the server cluster your query hits bounded by current load and the target service level for returning something. I wouldn't be surprised that your experience has recently degraded in this regard. The web is bigger every day.
- justinclift 5y agoUnfortunately "Recently degraded" means more like "fell off a cliff". :( The problem manifests itself again just now. Searching for these terms just gives bullshit results, instead of saying something useful like "no matches" or actually showing pages that meet what I asked for: "JBL" "ACTIVE-1" "Linux" Those specific search terms are due to my trying to work out whether the JBL ACTIVE-1 will work under linux. eg for driving my JBL LSR305/310S active monitors. There's a USB port, but no mention of Linux on the ACTIVE-1 product page. ;)
- brudgers 5y agoI understand the issue. I do a lot of non-IT (and non-consumer) related searching. Google quoting started getting worse for me probably ten years ago. And by "worse" I mean that before that, persistent searching could often "summon" better results. That is if I kept checking back with the same search terms, the results would tend to get better. Sometimes right away. Sometimes after a few hours or even days. In other words, poor searches would tend to get better and my mental model was that the index I was hitting would change based on my search and lack of interest in previous results. On the other hand, Google's results for technical topics (aka "things Google engineers value") has always been much better. I suspect this is because, search results require a lot of hand tuning of the "did you mean JPL" when "JBL" is in the search (not saying that's an actual example). --- I went over the dunes and sat by the Pacific after my last comment. I have a plausible theory for the recent dropoff. There's only so much hot memory on a server. Everything Google might return is there: Ads, the web page inverted index, AMP pages, etc. Google is pre-rendering more content for uses that are or look a lot like AMP. That means there's less space available for other things. The business decision is between ads and inverted index space. Your query isn't about a product you are about to buy. There's no money in it. And what are you going to do, use Bing? It's an engineered system. Economics are part of engineering. Google is a for profit business. You are a user, not a customer.