3 ms·

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

arnaudsm 2y ago
Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.
- stephantul 2y ago
  I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index. Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
- notyourwork 2y ago
  This reads like the “I could build Dropbox in a weekend”.
  - iterance 2y ago
    It's just files, how hard could it be?
    - rwaksmunski 2y ago
      No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.
  - arnaudsm 2y ago
    Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.
- marban 2y ago
  For news-only there's https://littleberg.com https://littleberg.com
marginalia_nu 2y ago
No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.
- arnaudsm 2y ago
  Big fan of your work Viktor, thanks for everything you build and how much you document it
- stephantul 2y ago
  I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient. This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
- tech234a 2y ago
  See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine. [1]: https://www.indexnow.org/ https://www.indexnow.org/
  - mdaniel 2y ago
    What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%20text%20key%20file%20at%20the%20root%20directory%20of%20your%20host https://www.indexnow.org/documentation#:~:text=Hosting%20a%2... 1: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml https://www.iana.org/assignments/well-known-uris/well-known-...
eitland 2y ago
The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages this it isn't trivial. But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.
- marginalia_nu 2y ago
  I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is. Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc. Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
  - deleted 2y ago
    [deleted]
  - l72 2y ago
    This is one of those things that I think is interesting about how "normal" people use the Internet. I am guessing they just always start with google. But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine. And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
    - alisonatwork 2y ago
      "Normal" people don't start with Google, not any more. They start with Facebook, Instagram, X, Reddit, Discord, Substack etc. That's exactly the problem, the world-wide web has devolved back into a collection of walled gardens like things were in the BBS era, except now the boards are all run by a handful of Silicon Valley billionaires instead of random nerds in your hometown.

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

arnaudsm 2y ago

Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.

stephantul 2y ago
I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index. Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
notyourwork 2y ago
This reads like the “I could build Dropbox in a weekend”.
- iterance 2y ago
  It's just files, how hard could it be?
  - rwaksmunski 2y ago
    No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.
- arnaudsm 2y ago
  Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.
marban 2y ago
For news-only there's https://littleberg.com https://littleberg.com

marginalia_nu 2y ago

No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.

arnaudsm 2y ago
Big fan of your work Viktor, thanks for everything you build and how much you document it
stephantul 2y ago
I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient. This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
tech234a 2y ago
See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine. [1]: https://www.indexnow.org/ https://www.indexnow.org/
- mdaniel 2y ago
  What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%20text%20key%20file%20at%20the%20root%20directory%20of%20your%20host https://www.indexnow.org/documentation#:~:text=Hosting%20a%2... 1: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml https://www.iana.org/assignments/well-known-uris/well-known-...

eitland 2y ago

The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages this it isn't trivial. But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.

marginalia_nu 2y ago
I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is. Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc. Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
- deleted 2y ago
  [deleted]
- l72 2y ago
  This is one of those things that I think is interesting about how "normal" people use the Internet. I am guessing they just always start with google. But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine. And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
  - alisonatwork 2y ago
    "Normal" people don't start with Google, not any more. They start with Facebook, Instagram, X, Reddit, Discord, Substack etc. That's exactly the problem, the world-wide web has devolved back into a collection of walled gardens like things were in the BBS era, except now the boards are all run by a handful of Silicon Valley billionaires instead of random nerds in your hometown.