4 ms·

Maybe the internet is just too big now. How can they search over trillions of documents using the same algorithm as before? Probably too expensive. Let alone h

Maybe the internet is just too big now. How can they search over trillions of documents using the same algorithm as before? Probably too expensive. Let alone how are they affording to index and crawl all those pages, which is still only a tiny fraction of the entire internet. Maybe search engines are a dying breed, and we'll have to rely on AI and word of mouth for information soon.

tyingq 4y ago
If I understand it right, it's not exactly the same algorithm. It's Pagerank, with lots more features and weights than in the past, run first. Then quite a lot of bolted on ML passes afterwards, each specialized for something different. One to suss out low quality content, one to identify low quality incoming links, one to boost the effect of brand/authority, and so on...many many more of these. Which apparently worked well for quite a long time, but also dangerous over time because you don't really know why something now ranks well, or doesn't. The best you can do is have human raters score various searches, tweak a bunch of weights/knobs, and try over and over again. Meaning, perhaps it's not that the internet is too big, but that the overall model is too big and disjointed to manage well. Also, if you're on this team, you're competing with the ad side that's continually pushing the organic results down the page, where they matter less. A lot of fussy tuning around "quaint hotel in Paris" seems wasted when the end user is 3 scroll events away from the organic results.
- PKop 4y ago
  This would be consistent with the internet being "too messy", if not too big generally. There is too much garbage to sift through, and spam trying to game the algorithm.
brokenkebaby 4y ago
I don't think it requires more CPU ops compared to GPT, or bigger bandwidth.
- Beaver117 4y ago
  GPT was trained on mostly precrawled data and books/code. Go calculate how much bandwidth it takes to crawl the whole internet
  - brokenkebaby 4y ago
    You need to do it anyway, if you're in business of internet search. The difference between PageRank, GPT, or whatever else algo is how you arrange results

tyingq 4y ago

If I understand it right, it's not exactly the same algorithm. It's Pagerank, with lots more features and weights than in the past, run first. Then quite a lot of bolted on ML passes afterwards, each specialized for something different. One to suss out low quality content, one to identify low quality incoming links, one to boost the effect of brand/authority, and so on...many many more of these. Which apparently worked well for quite a long time, but also dangerous over time because you don't really know why something now ranks well, or doesn't. The best you can do is have human raters score various searches, tweak a bunch of weights/knobs, and try over and over again. Meaning, perhaps it's not that the internet is too big, but that the overall model is too big and disjointed to manage well. Also, if you're on this team, you're competing with the ad side that's continually pushing the organic results down the page, where they matter less. A lot of fussy tuning around "quaint hotel in Paris" seems wasted when the end user is 3 scroll events away from the organic results.

PKop 4y ago
This would be consistent with the internet being "too messy", if not too big generally. There is too much garbage to sift through, and spam trying to game the algorithm.

brokenkebaby 4y ago

I don't think it requires more CPU ops compared to GPT, or bigger bandwidth.

Beaver117 4y ago
GPT was trained on mostly precrawled data and books/code. Go calculate how much bandwidth it takes to crawl the whole internet
- brokenkebaby 4y ago
  You need to do it anyway, if you're in business of internet search. The difference between PageRank, GPT, or whatever else algo is how you arrange results