4 ms·
"We don't have to have a Google and a Twitter and a Facebook." Many years ago, before Facebook or Twitter existed, I recall being promised that a project calle
by 1vuio0pswjnm7 3y ago
"We don't have to have a Google and a Twitter and a Facebook."
Many years ago, before Facebook or Twitter existed, I recall being promised that a project called "Nutch" would allow web users to crawl the web themselves.
Perhaps that promise is similar to the promises being made about "AI" today.
The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all.
- m-i-l 3y ago> ... a project called "Nutch" would allow web users to crawl the web themselves. Perhaps that promise is similar to the promises being made about "AI" today. The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all. Actually Nutch is used to produce the Common Crawl[0] and 60% of GPT-3's training data was Common Crawl[1], so in a way it is being used by a lot of web users today. I did very briefly look into the possibility of self-hosting a search based on the Common Crawl data, but the latest size is 390Tb, and given the largest consumer drives are around 10-14Tb, that would be a lot of drives and a lot of money. Plus by all accounts most of Common Crawl is poor quality spam type of content, which OpenAI, Google etc. try to filter out before using it to train their LLMs. That's why I don't think the solution is a search engine for the whole internet any more, but one which just indexes the relatively small minority of good bits. [0] https://commoncrawl.org/ https://commoncrawl.org/ [1] https://en.wikipedia.org/wiki/GPT-3 https://en.wikipedia.org/wiki/GPT-3
- marginalia_nu 3y agoCommon Crawl's data is a bit inflated due to how it's stored. WARC is the opposite of a compression format ;-) I think you are correct in that one of the primary problems faced in search is selecting interesting documents. I think I throw out like 75% of what I crawl because it's not very likely to ever be a good search result. You can probably filter it even more aggressively. It's a smart avenue to approach search not in the sense of "how can I make this computer cluster big enough to fit the data" but "how can I make this data small enough to fit my computer"..
- taskforcegemini 3y agospeaking as a sysadmin: random agressive crawlers are annoying and usually get blocked/filtered out. I think the best solution would be some sort of selfcrawler which uploads your site to a search provider. uploaders have to pay and bad uploads/seo-spam gets punished somehow (thus seo-abuse becomes expensive)
- mrkramer 3y ago>I think the best solution would be some sort of selfcrawler which uploads your site to a search provider. uploaders have to pay and bad uploads/seo-spam gets punished somehow (thus seo-abuse becomes expensive) I had the same idea; the website owners and the webmasters can crawl their own sites and work together with search engines in order to get revenue share from ads (sort of licensing deal, similar to YouTube partner revenue share program). But the problem is that the webmasters can not efficiently crawl their own sites because they do not have the money to do it every day like Google. Maybe we can have like 1 month old index for a search engine, sort of outdated and not that practical but still practical for some casual users. This idea would be better for a commercial web archiving service where website owners and webmasters crawl/archive their own website and license it to the web archiving service.
- 1vuio0pswjnm7 3y agoThink like a www user. Scale down, not up. For me, the best improvments that have occurred have been sitemaps XML. When paired with HTTP/1.1 pipelining I can peruse and download all pages that I want from a site in a single TCP connection. No CSS, Javascript, images, etc. (Won't find that cruft in Common Crawl either.) This is not "crawling", it's bulk text retrieval. It's starting with a list of URLs from a sitemap XML file and pipelining a series of pages, usually MIME type text/html, in a sequential order, into a compressed archive. Like the KC and the Sunshine Band song, that's the way I like it. Almost nothing on the www is universal and neither sitemaps nor pipelining are exceptions. Some sites do not even have a robots.txt. (One reason I like HTTP headers they are arguably one of the only universals on the www. All sites have them. Not much variation, either. If "Big Tech" has their way, who knows, that may change, too. HTTP/2 and HTTP/3 seems to give short shrift to HTTP headers. If I'm wrong about that, then that's good to hear.) Web search engines are opinionated. They are created by people who must make decisions about what to include and what exclude, what to favour and what to disfavour. Obviously, Google's opinion is influenced by advertisers and it hides behind a veil of secrecy as it stacks the deck in its own favour; the company completely abandoned the "search engine in the academic realm" idea Page and Brin wrote about in 1995. It's commercial, but in an insidious, passive-aggressive, "haha no one will notice" sort of way. Marginalia.nu (or wiby.me) might be one of the few non-commercial projects. Anyway, AFAIK, downloading sitemap XML and then HTML pages in bulk does not give so-called "tech" companies much data to use for advertising or other commercial purposes. It's not like sending one's www search queries to a "search engine".