10 ms·

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company building their own search engine from scratch?

stephantul 2y ago
Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.
- arnaudsm 2y ago
  Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.
  - stephantul 2y ago
    I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index. Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
  - notyourwork 2y ago
    This reads like the “I could build Dropbox in a weekend”.
    - iterance 2y ago
      It's just files, how hard could it be?
      - rwaksmunski 2y ago
        No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.
    - arnaudsm 2y ago
      Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.
  - marban 2y ago
    For news-only there's https://littleberg.com https://littleberg.com
- marginalia_nu 2y ago
  No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.
  - arnaudsm 2y ago
    Big fan of your work Viktor, thanks for everything you build and how much you document it
  - stephantul 2y ago
    I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient. This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
  - tech234a 2y ago
    See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine. [1]: https://www.indexnow.org/ https://www.indexnow.org/
    - mdaniel 2y ago
      What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%20text%20key%20file%20at%20the%20root%20directory%20of%20your%20host https://www.indexnow.org/documentation#:~:text=Hosting%20a%2... 1: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml https://www.iana.org/assignments/well-known-uris/well-known-...
- eitland 2y ago
  The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages this it isn't trivial. But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.
  - marginalia_nu 2y ago
    I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is. Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc. Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
    - deleted 2y ago
      [deleted]
    - l72 2y ago
      This is one of those things that I think is interesting about how "normal" people use the Internet. I am guessing they just always start with google. But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine. And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
      - alisonatwork 2y ago
        "Normal" people don't start with Google, not any more. They start with Facebook, Instagram, X, Reddit, Discord, Substack etc. That's exactly the problem, the world-wide web has devolved back into a collection of walled gardens like things were in the BBS era, except now the boards are all run by a handful of Silicon Valley billionaires instead of random nerds in your hometown.
pseudocomposer 2y ago
I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know. https://kagi.com https://kagi.com
- malfist 2y ago
  As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking
  - chabad360 2y ago
    No they do not, they have their own indexes. https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
    - rkangel 2y ago
      That page includes the text "Our search results also include anonymized API calls to all major search result providers worldwide". They source results from lots of places including Google. One way that you can confirm this is to search for something that only appears in a recent Reddit post. Google has done a deal with Reddit that they're the only company allowed to index Reddit since the summer. DuckDuckGo gets no answers if you specify only results from the last week: https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com&t=h_&df=w https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com... Kagi is fine if you do the same: https://kagi.com/search?q=caleb+williams+site%3Areddit.com&dr=2 https://kagi.com/search?q=caleb+williams+site%3Areddit.com&d... edit: I don't think this is a bad thing for Kagi. I'm a very happy subscriber, and it's nice for me that I still get results from Reddit. They're very useful!
      - immibis 2y ago
        Note that due to adversarial interoperability, search engines other than Google can scrape Reddit if they try hard enough. A rotating residential proxy subscription, while pricey, likely still costs orders of magnitude less than what Google paid. The same goes for Stack Overflow. You can also DIY by getting a handful of SIM cards. CGNAT, usually a scourge, works in your favour for this application since Reddit can't tell the difference between you loading 10000 pages and 10000 people on your ISP loading one page each (depending on the ISP)
onli 2y ago
Brave search is doing it properly. Bought tech, but still.
- vishnumohandas 2y ago
  > Bought tech Could you please elaborate?
  - onli 2y ago
    Brave search is a continuation of cliqz. A German company that developed a proper search engine, with an independent index. They shut down, but the tech got sold off. Cliqz was the first time for me that a Google alternative actually worked really well - and it, or now brave search, is what parent was asking for :)
    - prophesi 2y ago
      Yet we're still back to Larry Page and Sergey Brin's conclusion in their "The Anatomy of a Large-Scale Hypertextual Web Search Engine" research paper[0]: > We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers. Brave Search Premium hasn't been around nearly as long as their free tier serving ads, and I'm not confident this conflict of interest is gone. Having independent indexes is a win regardless though. [0] https://snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom...
      - BrendanEich 2y ago
        Brave Search Premium launched before we showed any search ads: https://x.com/brave/status/1466510541128548362 https://x.com/brave/status/1466510541128548362
- miyuru 2y ago
  very recently found about brave goggles. amazing way to give control to users, for ex: blocking pinterest or searching domains popular with HN or own list. https://search.brave.com/goggles/discover https://search.brave.com/goggles/discover
throawayonthe 2y ago
[dead]
unionpivo 2y ago
Because nowdays more than ever content you need is in silos. Your facebooks/twiters/instagram/stack overflow/reddit ... And they all have limited expensive api's, and have bulk scrapping detection. Sure you can clobber together something that will work for a while, but you can't runn a buissness on that. Aditionaly most paywalled sites (like news) explicitly whitlist google and bing, and if someone cretes new site, they do the same. As an upstart you would have to reach out to them to get them to whitelist you. and you would need to do it not only in USA but globaly. Anothe problem is cloudflare and other cdns/web firewalls, so even trying to index mom and pops blog site could be problematic. An d most of the mom and pop blogs are nowdays on som ploging platform that is just another silo. Now that i think about it, cloudflare might be in a good position to do it. The AI hype and scraping for content to feed the models have increased dificulty for anyone new to start new index.
- arnaudsm 2y ago
  This is the best (and saddest) answer. LLMs break the social contract of the internet, we're in a feudalisation process. The decentralized nature of the internet was amazing for businesses, and monopolization could ruin the space and slow innovation down significantly.
  - ColinHayhurst 2y ago
    > LLMs break the social contract of the internet The legal concept of fair usage has and is being challenged, and will best tested in court. Is the Golden Age of Fair Use Over? Maybe [0]. [0] https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-use-over.html https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-us...
  - tensor 2y ago
    While LLMs have accelerated, it, it was already the case that silos were blocking non-Google and non-Bing results before LLMs. LLMs have only made existing problems of the web worse, but they were problems before LLMs too and banning LLMs won't fix the core issues of silos and misinformation.
- 2y ago
jasode 2y ago
>competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. [...] CommonCrawl text-only is ~100TB, Those example 3 bullet points of today's improved 2024 computing power you list isn't even enough to process Google's scale 14 years ago in 2010 when the search index was 100+ petabytes: https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... A modest homegrown tech stack of 2024 can maybe compete with a smaller Google circa ~1998 but that thought experiment is handicapping Google's current state-of-the-art. Instead, we have to compare OSS-today vs Google-today. There's still a big delta gap between the 2024 OSS tech stack and Google's internal 2024 tech stack. E.g. for all the billions Microsoft spent on Bing, there are still some queries that I noticed Google was better at. Google found more pages of obscure people I was researching (obituaries, etc). But Bing had the edge when I was looking up various court cases with docket #s. The internet is now so big that even billion dollar search engines can't get to all of it. Each has blindspots. I have to use both search engines every single day.
- arnaudsm 2y ago
  I was talking about text-only, filtered and deduped content. Most of Google's 100PB is picture and video. Filtering the spam and deduping the content helped Google reduce the ~50B page index in 2012 to ~<10B today.
  - deleted 2y ago
    [deleted]
  - the-rc 2y ago
    Where are these figures coming from?
  - Larrikin 2y ago
    But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s and all the content you just threw away as irrelevant actually contains information I am looking for. There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.
ColinHayhurst 2y ago
We are @mojeek
ColinHayhurst 2y ago
Mostly because Google bought, developed, acquired or effectively control all the major distribution points with default placement deals: eg Apple, Samsung, Chrome, Android, Firefox. In time remedies are coming though, in the antitrust case lost versus DoJ. Another major factor is that building a search index and algorithms that searches across billions of pages with good enough latency is very hard. Easy enough for 10s of millions scale search but a different challenge for billions. Some claim(ed) click-query data is needed at scale, and are hoping for that remedy. Our take is what is the point of replicating Google. Anyway, will this data be free or low cost? You know the answer. Cloud infrastructure is very expensive. We save massively on costs by building our own servers, but that means capital outlay.
- Eisenstein 2y ago
  Remember that MS lost the anti-trust too, and after the presidential transfer in 2000, they pretty much dropped it with few consequences for MS.
  - gary_0 2y ago
    If I had to guess I'd say history is going to repeat itself and Google will also escape any significant consequences.
    - immibis 2y ago
      Maybe, maybe not. Remember that Google is "woke" - an enemy of the faction that now finds itself in power. They might continue the lawsuit to set an example.
1vuio0pswjnm7 2y ago
"CommonCrawl [being] text-only is only ~100TB, and can fit on my home server." Are any individual users downloading CC for potential use in the future? It may seem like a non-trivial task to process ~100TB at home today but in the future processing this amount of data will likely seem trivial. CC data is available for download to anyone but, to me, it appears only so-called "tech" companies and "researchers" are grabbing a copy. Many years ago I began storing copies of the publicly-available com. and net. zonefiles from Verisign. At the time it was infeasible for me to try to serve multi-GB zonefiles on the local network at home. Today, it's feasible. And in the future it will be even easier. NB. I am not employed by a so-called "tech" company. I store this data for personal, non-commercial use.
- 1024core 2y ago
  Is Common Crawl updated frequently?
  - astonex 2y ago
    They do monthly-ish releases https://index.commoncrawl.org/ https://index.commoncrawl.org/
nijave 2y ago
I'm guessing there's no money in it unless you glue an ad machine to the side and use search to drive advertising.
kev009 2y ago
There is probably room for one or five lifestyle businesses but convincing venture capital to drop the megabux to go big would be a feat and eventually land at some sub-optimal state anyway. Finding some hack to democratize&decentralize the indexing and expensive processes like JavaScript interpretation, image interpretation, OCR, etc is an open angle and even an avenue for "Web3" to offload the cost. But you will ultimately want the core search and index on a tighter knit cluster (many computers physically close to one another for speed of light reasons, although you can have N of these clusters) for performance so it's a hard nut to crack for making something equitable for both the developers and any prospectors and safe from various takeovers and enshitifies. Let us know if you know a way.
- zzo38computer 2y ago
  I would want to use a search engine that does not perform JavaScript interpretation, image interpretation, OCR, etc. (This is not the same as excluding web pages with JavaScripts from the search results. They would still be included but only indexed by whatever text is available without JavaScripts; if there isn't any such text, then they should be excluded. This would also apply if it is only pictures, video, etc and no text, then they also cannot be indexed, whether or not they have JavaScripts.)
  - kev009 2y ago
    Luckily none of these things are mutually exclusive. Thanks to all the SPA idiocy you will miss enough content to matter if you have zero JS interpretation, so you would want to let the user choose which indexes they want for a query because sometimes you need these other resource types to answer the query.
analog31 2y ago
It occurs to me that competing with Google on their home turf and at their scale might be impossible. But doing what Google was good at, when they just a search engine, might not be all that much harder today than it was back then. And it may fly "under the radar" of Google's current business priorities.
- flir 2y ago
  I'm thinking about how Etsy took "the good bit" of Ebay and ran with it. Or how tyre-and-exhaust shops take "the profitable bit" of being a mechanic and run with it. I know there's a name for this, but it eludes me.
  - analog31 2y ago
    It could be they also took the bit that eBay abandoned. There was a point when eBay announced to the business press that they were going to de-emphasize the "America's yard sale" aspect, and shift towards being a regular e-commerce site like Amazon.
dmafreezone 2y ago
Isn’t it obvious? Look around. Hackers have been stamped out even on hacker news. Now it’s all FAANG lifers and MBA or VC types trying their hand at grifting. Nothing good comes of that. Whoever is making the next thing “for real” just gets acquired and shut down. Moreover, get out of your echo chamber and you’ll see that for a majority of humanity Google is the internet. You have to supplant the utility, not just the brand. Most businesses cannot handle the latter let alone the former. If you want something to replace Google, you have to think about replacing the internet itself. But not many are that bold.
- lazystar 2y ago
  > Whoever is making the next thing “for real” just gets acquired and shut down. why do you think amazon's trying to lay people off? they want them to create new startups that they can acquire.
Animats 2y ago
Because most of the good sites won't let you crawl them any more, unless you're Google.
- immibis 2y ago
  Crawl them anyway. Adversarial interoperability is the name of the game.
weitendorf 2y ago
To properly compete with Google search you have to: 1. Create your own index/crawl and true search engine on top of it, rather than delegating that back to bing or Google. In doing so, solve or address all the related problems like SEO/spam, and cover the “long tail” (expensive part) 2. Monetize it somehow to afford all of the personnel/hardware/software costs. Probably with your own version of Google’s ad products. 3. Solve user acquisition. Note that Google’s user acquisition process involves multiB/yr contracts with Apple and huge deals with other vendors who control the “top of funnel” like Mozilla Firefox, and that this is the sole purpose of chrome/android/chromebook/etc who you’ll never be able to make a deal with. You will probably at the very minimum need to implement your own platform (device, OS, browser alone probably won’t cut it). 4. Solve the bootstrap problem of getting enough users for people to care about letting you index their site, without initially being able to index a lot of sites, because you are not important enough. 5. Somehow pull all of this off in plain site of Google (this would take many years to build both technically and in terms of users) without them being able to properly fend you off 6. Somehow pull all of this off in spite of the web/web search seeming like it’s going to die off or fade into irrelevance OR you can dedicate a decade of your life to something more likely to succeed.
- arrakark 2y ago
  I'd like to add to your #5. If Google deems you a legitimate threat, then they can just de-crapify their own search for a bit by going back to their old algo. It's extremely easy for them to fight back.
  - Maakuth 2y ago
    Is that true though? I think a bit part of the algorithm evolution is to combat undesirable forms of SEO. I would rolling back the algorithm to some older generation would make the results quite spammy.
  - johnisgood 2y ago
    That would be a victory for users.
xbmcuser 2y ago
Google ingests almost the whole public web almost every day. I don't see any startup competing with them they might come out with a great algo or something but will need the infrastructure and huge investments to compete. Even then after using Google Gemini subscription the last few months I think the problem is not Google search rather the web ecosystem as if google gives me most of the answers without having to click any link you and I might be happy but billion people living off those links directly or indirectly won't be happy.
r721 2y ago
>Why is no company building their own search engine from scratch? Here's one list: >A look at search engines with their own indexes https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ https://seirdy.one/posts/2021/03/10/search-engines-with-own-... HN discussions: https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=31820149 https://news.ycombinator.com/item?id=31820149 https://news.ycombinator.com/item?id=40626011 https://news.ycombinator.com/item?id=40626011
ezekiel68 2y ago
Kagi is not one of these and I love it enough to pay for it. It has kept the semi-"advanced" features of respecting the negative sign and emphasizing results that match quoted terms in the search query.
gurgunday 2y ago
Brave Search ONLY uses their own index and the results are really good! I highly recommend it.
- HKH2 2y ago
  Brave image search is not that good though.
whiplash451 2y ago
Because “search” in 2024 is much more than a search engine. It takes billions of dollars of investment per year to build something competitive to Google head-on. OpenAI may have a shot with their chat.com The only viable option when starting small is to grow in the cracks of Google with features that users really love that Google won’t provide.
LunaSea 2y ago
A single CommonCrawl dump might be 100TB but that represents less than 1% of the Internet. CommonCrawl crawls news parts of the Internet every month / trimester and there is little overlap between crawl dumps.
gargan 2y ago
Even ChatGPT Search was revealed to be a Bing wrapper, this meme summed it up - https://ibb.co/8csc3gv https://ibb.co/8csc3gv
- Spivak 2y ago
  Because the hard part isn't the compute, vector dbs, or whatever. It's the huge evergreen index of the whole internet. Getting over the hump of "every site lets your crawler work, gives good results to it, and bypass paywalls" is a massive barrier.
thomvaughan 2y ago
It's worth noting that building a solid search index needs more than just text. The full Common Crawl archive includes metadata, links, page structure, and other signals that are needed for relevance and ranking, and to date is over 7 PiB, so that's rather more than tends to fit on home servers
z3ncyberpunk 2y ago
Damn must be nice to be rich enough to be able to casually have 100TB of space for such projects.

stephantul 2y ago

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

arnaudsm 2y ago
Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.
- stephantul 2y ago
  I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index. Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
- notyourwork 2y ago
  This reads like the “I could build Dropbox in a weekend”.
  - iterance 2y ago
    It's just files, how hard could it be?
    - rwaksmunski 2y ago
      No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.
  - arnaudsm 2y ago
    Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.
- marban 2y ago
  For news-only there's https://littleberg.com https://littleberg.com
marginalia_nu 2y ago
No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.
- arnaudsm 2y ago
  Big fan of your work Viktor, thanks for everything you build and how much you document it
- stephantul 2y ago
  I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient. This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
- tech234a 2y ago
  See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine. [1]: https://www.indexnow.org/ https://www.indexnow.org/
  - mdaniel 2y ago
    What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%20text%20key%20file%20at%20the%20root%20directory%20of%20your%20host https://www.indexnow.org/documentation#:~:text=Hosting%20a%2... 1: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml https://www.iana.org/assignments/well-known-uris/well-known-...
eitland 2y ago
The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages this it isn't trivial. But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.
- marginalia_nu 2y ago
  I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is. Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc. Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
  - deleted 2y ago
    [deleted]
  - l72 2y ago
    This is one of those things that I think is interesting about how "normal" people use the Internet. I am guessing they just always start with google. But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine. And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
    - alisonatwork 2y ago
      "Normal" people don't start with Google, not any more. They start with Facebook, Instagram, X, Reddit, Discord, Substack etc. That's exactly the problem, the world-wide web has devolved back into a collection of walled gardens like things were in the BBS era, except now the boards are all run by a handful of Silicon Valley billionaires instead of random nerds in your hometown.

pseudocomposer 2y ago

I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know. https://kagi.com https://kagi.com

malfist 2y ago
As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking
- chabad360 2y ago
  No they do not, they have their own indexes. https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
  - rkangel 2y ago
    That page includes the text "Our search results also include anonymized API calls to all major search result providers worldwide". They source results from lots of places including Google. One way that you can confirm this is to search for something that only appears in a recent Reddit post. Google has done a deal with Reddit that they're the only company allowed to index Reddit since the summer. DuckDuckGo gets no answers if you specify only results from the last week: https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com&t=h_&df=w https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com... Kagi is fine if you do the same: https://kagi.com/search?q=caleb+williams+site%3Areddit.com&dr=2 https://kagi.com/search?q=caleb+williams+site%3Areddit.com&d... edit: I don't think this is a bad thing for Kagi. I'm a very happy subscriber, and it's nice for me that I still get results from Reddit. They're very useful!
    - immibis 2y ago
      Note that due to adversarial interoperability, search engines other than Google can scrape Reddit if they try hard enough. A rotating residential proxy subscription, while pricey, likely still costs orders of magnitude less than what Google paid. The same goes for Stack Overflow. You can also DIY by getting a handful of SIM cards. CGNAT, usually a scourge, works in your favour for this application since Reddit can't tell the difference between you loading 10000 pages and 10000 people on your ISP loading one page each (depending on the ISP)

onli 2y ago

Brave search is doing it properly. Bought tech, but still.

vishnumohandas 2y ago
> Bought tech Could you please elaborate?
- onli 2y ago
  Brave search is a continuation of cliqz. A German company that developed a proper search engine, with an independent index. They shut down, but the tech got sold off. Cliqz was the first time for me that a Google alternative actually worked really well - and it, or now brave search, is what parent was asking for :)
  - prophesi 2y ago
    Yet we're still back to Larry Page and Sergey Brin's conclusion in their "The Anatomy of a Large-Scale Hypertextual Web Search Engine" research paper[0]: > We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers. Brave Search Premium hasn't been around nearly as long as their free tier serving ads, and I'm not confident this conflict of interest is gone. Having independent indexes is a win regardless though. [0] https://snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom...
    - BrendanEich 2y ago
      Brave Search Premium launched before we showed any search ads: https://x.com/brave/status/1466510541128548362 https://x.com/brave/status/1466510541128548362
miyuru 2y ago
very recently found about brave goggles. amazing way to give control to users, for ex: blocking pinterest or searching domains popular with HN or own list. https://search.brave.com/goggles/discover https://search.brave.com/goggles/discover

throawayonthe 2y ago

[dead]

unionpivo 2y ago

Because nowdays more than ever content you need is in silos. Your facebooks/twiters/instagram/stack overflow/reddit ... And they all have limited expensive api's, and have bulk scrapping detection. Sure you can clobber together something that will work for a while, but you can't runn a buissness on that. Aditionaly most paywalled sites (like news) explicitly whitlist google and bing, and if someone cretes new site, they do the same. As an upstart you would have to reach out to them to get them to whitelist you. and you would need to do it not only in USA but globaly. Anothe problem is cloudflare and other cdns/web firewalls, so even trying to index mom and pops blog site could be problematic. An d most of the mom and pop blogs are nowdays on som ploging platform that is just another silo. Now that i think about it, cloudflare might be in a good position to do it. The AI hype and scraping for content to feed the models have increased dificulty for anyone new to start new index.

arnaudsm 2y ago
This is the best (and saddest) answer. LLMs break the social contract of the internet, we're in a feudalisation process. The decentralized nature of the internet was amazing for businesses, and monopolization could ruin the space and slow innovation down significantly.
- ColinHayhurst 2y ago
  > LLMs break the social contract of the internet The legal concept of fair usage has and is being challenged, and will best tested in court. Is the Golden Age of Fair Use Over? Maybe [0]. [0] https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-use-over.html https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-us...
- tensor 2y ago
  While LLMs have accelerated, it, it was already the case that silos were blocking non-Google and non-Bing results before LLMs. LLMs have only made existing problems of the web worse, but they were problems before LLMs too and banning LLMs won't fix the core issues of silos and misinformation.
2y ago

jasode 2y ago

>competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. [...] CommonCrawl text-only is ~100TB, Those example 3 bullet points of today's improved 2024 computing power you list isn't even enough to process Google's scale 14 years ago in 2010 when the search index was 100+ petabytes: https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... A modest homegrown tech stack of 2024 can maybe compete with a smaller Google circa ~1998 but that thought experiment is handicapping Google's current state-of-the-art. Instead, we have to compare OSS-today vs Google-today. There's still a big delta gap between the 2024 OSS tech stack and Google's internal 2024 tech stack. E.g. for all the billions Microsoft spent on Bing, there are still some queries that I noticed Google was better at. Google found more pages of obscure people I was researching (obituaries, etc). But Bing had the edge when I was looking up various court cases with docket #s. The internet is now so big that even billion dollar search engines can't get to all of it. Each has blindspots. I have to use both search engines every single day.

arnaudsm 2y ago
I was talking about text-only, filtered and deduped content. Most of Google's 100PB is picture and video. Filtering the spam and deduping the content helped Google reduce the ~50B page index in 2012 to ~<10B today.
- deleted 2y ago
  [deleted]
- the-rc 2y ago
  Where are these figures coming from?
- Larrikin 2y ago
  But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s and all the content you just threw away as irrelevant actually contains information I am looking for. There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.

ColinHayhurst 2y ago

We are @mojeek

ColinHayhurst 2y ago

Mostly because Google bought, developed, acquired or effectively control all the major distribution points with default placement deals: eg Apple, Samsung, Chrome, Android, Firefox. In time remedies are coming though, in the antitrust case lost versus DoJ. Another major factor is that building a search index and algorithms that searches across billions of pages with good enough latency is very hard. Easy enough for 10s of millions scale search but a different challenge for billions. Some claim(ed) click-query data is needed at scale, and are hoping for that remedy. Our take is what is the point of replicating Google. Anyway, will this data be free or low cost? You know the answer. Cloud infrastructure is very expensive. We save massively on costs by building our own servers, but that means capital outlay.

Eisenstein 2y ago
Remember that MS lost the anti-trust too, and after the presidential transfer in 2000, they pretty much dropped it with few consequences for MS.
- gary_0 2y ago
  If I had to guess I'd say history is going to repeat itself and Google will also escape any significant consequences.
  - immibis 2y ago
    Maybe, maybe not. Remember that Google is "woke" - an enemy of the faction that now finds itself in power. They might continue the lawsuit to set an example.

1vuio0pswjnm7 2y ago

"CommonCrawl [being] text-only is only ~100TB, and can fit on my home server." Are any individual users downloading CC for potential use in the future? It may seem like a non-trivial task to process ~100TB at home today but in the future processing this amount of data will likely seem trivial. CC data is available for download to anyone but, to me, it appears only so-called "tech" companies and "researchers" are grabbing a copy. Many years ago I began storing copies of the publicly-available com. and net. zonefiles from Verisign. At the time it was infeasible for me to try to serve multi-GB zonefiles on the local network at home. Today, it's feasible. And in the future it will be even easier. NB. I am not employed by a so-called "tech" company. I store this data for personal, non-commercial use.

1024core 2y ago
Is Common Crawl updated frequently?
- astonex 2y ago
  They do monthly-ish releases https://index.commoncrawl.org/ https://index.commoncrawl.org/

nijave 2y ago

I'm guessing there's no money in it unless you glue an ad machine to the side and use search to drive advertising.

kev009 2y ago

There is probably room for one or five lifestyle businesses but convincing venture capital to drop the megabux to go big would be a feat and eventually land at some sub-optimal state anyway. Finding some hack to democratize&decentralize the indexing and expensive processes like JavaScript interpretation, image interpretation, OCR, etc is an open angle and even an avenue for "Web3" to offload the cost. But you will ultimately want the core search and index on a tighter knit cluster (many computers physically close to one another for speed of light reasons, although you can have N of these clusters) for performance so it's a hard nut to crack for making something equitable for both the developers and any prospectors and safe from various takeovers and enshitifies. Let us know if you know a way.

zzo38computer 2y ago
I would want to use a search engine that does not perform JavaScript interpretation, image interpretation, OCR, etc. (This is not the same as excluding web pages with JavaScripts from the search results. They would still be included but only indexed by whatever text is available without JavaScripts; if there isn't any such text, then they should be excluded. This would also apply if it is only pictures, video, etc and no text, then they also cannot be indexed, whether or not they have JavaScripts.)
- kev009 2y ago
  Luckily none of these things are mutually exclusive. Thanks to all the SPA idiocy you will miss enough content to matter if you have zero JS interpretation, so you would want to let the user choose which indexes they want for a query because sometimes you need these other resource types to answer the query.

analog31 2y ago

It occurs to me that competing with Google on their home turf and at their scale might be impossible. But doing what Google was good at, when they just a search engine, might not be all that much harder today than it was back then. And it may fly "under the radar" of Google's current business priorities.

flir 2y ago
I'm thinking about how Etsy took "the good bit" of Ebay and ran with it. Or how tyre-and-exhaust shops take "the profitable bit" of being a mechanic and run with it. I know there's a name for this, but it eludes me.
- analog31 2y ago
  It could be they also took the bit that eBay abandoned. There was a point when eBay announced to the business press that they were going to de-emphasize the "America's yard sale" aspect, and shift towards being a regular e-commerce site like Amazon.

dmafreezone 2y ago

Isn’t it obvious? Look around. Hackers have been stamped out even on hacker news. Now it’s all FAANG lifers and MBA or VC types trying their hand at grifting. Nothing good comes of that. Whoever is making the next thing “for real” just gets acquired and shut down. Moreover, get out of your echo chamber and you’ll see that for a majority of humanity Google is the internet. You have to supplant the utility, not just the brand. Most businesses cannot handle the latter let alone the former. If you want something to replace Google, you have to think about replacing the internet itself. But not many are that bold.

lazystar 2y ago
> Whoever is making the next thing “for real” just gets acquired and shut down. why do you think amazon's trying to lay people off? they want them to create new startups that they can acquire.

Animats 2y ago

Because most of the good sites won't let you crawl them any more, unless you're Google.

immibis 2y ago
Crawl them anyway. Adversarial interoperability is the name of the game.

weitendorf 2y ago

To properly compete with Google search you have to: 1. Create your own index/crawl and true search engine on top of it, rather than delegating that back to bing or Google. In doing so, solve or address all the related problems like SEO/spam, and cover the “long tail” (expensive part) 2. Monetize it somehow to afford all of the personnel/hardware/software costs. Probably with your own version of Google’s ad products. 3. Solve user acquisition. Note that Google’s user acquisition process involves multiB/yr contracts with Apple and huge deals with other vendors who control the “top of funnel” like Mozilla Firefox, and that this is the sole purpose of chrome/android/chromebook/etc who you’ll never be able to make a deal with. You will probably at the very minimum need to implement your own platform (device, OS, browser alone probably won’t cut it). 4. Solve the bootstrap problem of getting enough users for people to care about letting you index their site, without initially being able to index a lot of sites, because you are not important enough. 5. Somehow pull all of this off in plain site of Google (this would take many years to build both technically and in terms of users) without them being able to properly fend you off 6. Somehow pull all of this off in spite of the web/web search seeming like it’s going to die off or fade into irrelevance OR you can dedicate a decade of your life to something more likely to succeed.

arrakark 2y ago
I'd like to add to your #5. If Google deems you a legitimate threat, then they can just de-crapify their own search for a bit by going back to their old algo. It's extremely easy for them to fight back.
- Maakuth 2y ago
  Is that true though? I think a bit part of the algorithm evolution is to combat undesirable forms of SEO. I would rolling back the algorithm to some older generation would make the results quite spammy.
- johnisgood 2y ago
  That would be a victory for users.

xbmcuser 2y ago

Google ingests almost the whole public web almost every day. I don't see any startup competing with them they might come out with a great algo or something but will need the infrastructure and huge investments to compete. Even then after using Google Gemini subscription the last few months I think the problem is not Google search rather the web ecosystem as if google gives me most of the answers without having to click any link you and I might be happy but billion people living off those links directly or indirectly won't be happy.

r721 2y ago

>Why is no company building their own search engine from scratch? Here's one list: >A look at search engines with their own indexes https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ https://seirdy.one/posts/2021/03/10/search-engines-with-own-... HN discussions: https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=31820149 https://news.ycombinator.com/item?id=31820149 https://news.ycombinator.com/item?id=40626011 https://news.ycombinator.com/item?id=40626011

ezekiel68 2y ago

Kagi is not one of these and I love it enough to pay for it. It has kept the semi-"advanced" features of respecting the negative sign and emphasizing results that match quoted terms in the search query.

gurgunday 2y ago

Brave Search ONLY uses their own index and the results are really good! I highly recommend it.

HKH2 2y ago
Brave image search is not that good though.

whiplash451 2y ago

Because “search” in 2024 is much more than a search engine. It takes billions of dollars of investment per year to build something competitive to Google head-on. OpenAI may have a shot with their chat.com The only viable option when starting small is to grow in the cracks of Google with features that users really love that Google won’t provide.

LunaSea 2y ago

A single CommonCrawl dump might be 100TB but that represents less than 1% of the Internet. CommonCrawl crawls news parts of the Internet every month / trimester and there is little overlap between crawl dumps.

gargan 2y ago

Even ChatGPT Search was revealed to be a Bing wrapper, this meme summed it up - https://ibb.co/8csc3gv https://ibb.co/8csc3gv

Spivak 2y ago
Because the hard part isn't the compute, vector dbs, or whatever. It's the huge evergreen index of the whole internet. Getting over the hump of "every site lets your crawler work, gives good results to it, and bypass paywalls" is a massive barrier.

thomvaughan 2y ago

It's worth noting that building a solid search index needs more than just text. The full Common Crawl archive includes metadata, links, page structure, and other signals that are needed for relevance and ranking, and to date is over 7 PiB, so that's rather more than tends to fit on home servers

z3ncyberpunk 2y ago

Damn must be nice to be rich enough to be able to casually have 100TB of space for such projects.