5 ms·

Hey, your project is awesome, but you seem to be the only non-big3 crawler who doesn't encounter this problem and call attention to it [1]. Maybe because you (

Hey, your project is awesome, but you seem to be the only non-big3 crawler who doesn't encounter this problem and call attention to it [1]. Maybe because you (awesomely) focus on low/no-javascript, text-heavy sites? Much of the most obnoxious antibot stuff (reCAPTCHA, CDN "AI" throttling, a lot of what cloudflare does, etc) is javascript-based. Maybe I'm totally wrong here. If so, you really should make your crawl corpus publicly available. Trickle it into one of the public clouds and charge people a few microcents per query. You'll get to retire early, knowing you've done a good deed for human technological progress. > There are websites that give you different results depending on what time of day it is, or where you are from. That has been true since at least 1993 [2], before there were search engines! > The hard problem of search engines is to find a good subset of the internet for some definition of good This really is a case of the tech industry trying to redefine "search" in a way that gives them editorial power. Librarians don't do that sort of thing. We could learn a lot from their integrity. Or we could just stop calling it "search", because that's not what it is anymore. For example, that box at the top of amazon.com is most definitely not a search box anymore; it's a sales-lead-generator box. The button should be marked "try to sell me shit which is not necessarily related to these words" instead of "search". [1] off the top of my head: archive.is/archive.today, gigablast. I'm actually having trouble remembering any still-operating whole-web crawlers besides Google, Bing, and Yandex. [2] https://en.wikipedia.org/wiki/Common_Gateway_Interface https://en.wikipedia.org/wiki/Common_Gateway_Interface

ColinHayhurst 5y ago
Some folks think it's a very big issue and have released data on their findings about robots.txt files: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/ We see some problems but as an established, and respectful crawler, we are usually able to overcome most blocks at Mojeek; a international web search engine with no tracking (currently 4.3 billion pages)
- ringworld 5y ago
  I learned of Mojeek via searX use, it has found corners of the Internet which the big 3 have ignored with the same search queries; I find it very useful to balance out all the popular results. Kudos to your team.
marginalia_nu 5y ago
I'm aware that CGI has been around for a long time, but it was bog slow since it forked new processes to deal with every call, it wasn't really feasible to build the sort of dynamic websites we do today until stuff like PHP entered the market. The relative proportion of static files versus dynamic web content have probably reversed. > This really is a case of the tech industry trying to redefine "search" in a way that gives them editorial power. This isn't a new development at all. The signal to noise ratio problem goes back to the late '90s, and ruthlessly removing noise was arguably one of the moves that allowed Google to break through. Unfortunately they seem to have forgotten why they became a success. From http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html > [...] In 1994, some people believed that a complete search index would make it possible to find anything easily. [...] However, the Web of 1997 is quite different. Anyone who has used a search engine recently, can readily testify that the completeness of the index is not the only factor in the quality of search results. -- It's difficult to conceptualize these numbers, but we can use English Wikipedia as a measuring stick. It has of the order ten million pages, and covers most topics you would ever want to read about. Maybe there are some niche things here and there that are missing, but overall, it's a relatively complete coverage of human ideas and interests. How many English Wikipedia Units do you need, for your search engine to feel complete, assuming 100% signal and 0% noise? Do you need one? Ten? A hundred? A "big search engine" crawl is over a hundred thousand English Wikipedia Units. How much of that is interesting? How much has been written by humans, has ever been looked at by humans?

ColinHayhurst 5y ago

Some folks think it's a very big issue and have released data on their findings about robots.txt files: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/ We see some problems but as an established, and respectful crawler, we are usually able to overcome most blocks at Mojeek; a international web search engine with no tracking (currently 4.3 billion pages)

ringworld 5y ago
I learned of Mojeek via searX use, it has found corners of the Internet which the big 3 have ignored with the same search queries; I find it very useful to balance out all the popular results. Kudos to your team.

marginalia_nu 5y ago

I'm aware that CGI has been around for a long time, but it was bog slow since it forked new processes to deal with every call, it wasn't really feasible to build the sort of dynamic websites we do today until stuff like PHP entered the market. The relative proportion of static files versus dynamic web content have probably reversed. > This really is a case of the tech industry trying to redefine "search" in a way that gives them editorial power. This isn't a new development at all. The signal to noise ratio problem goes back to the late '90s, and ruthlessly removing noise was arguably one of the moves that allowed Google to break through. Unfortunately they seem to have forgotten why they became a success. From http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html > [...] In 1994, some people believed that a complete search index would make it possible to find anything easily. [...] However, the Web of 1997 is quite different. Anyone who has used a search engine recently, can readily testify that the completeness of the index is not the only factor in the quality of search results. -- It's difficult to conceptualize these numbers, but we can use English Wikipedia as a measuring stick. It has of the order ten million pages, and covers most topics you would ever want to read about. Maybe there are some niche things here and there that are missing, but overall, it's a relatively complete coverage of human ideas and interests. How many English Wikipedia Units do you need, for your search engine to feel complete, assuming 100% signal and 0% noise? Do you need one? Ten? A hundred? A "big search engine" crawl is over a hundred thousand English Wikipedia Units. How much of that is interesting? How much has been written by humans, has ever been looked at by humans?