2 ms·
Think like a www user. Scale down, not up. For me, the best improvments that have occurred have been sitemaps XML. When paired with HTTP/1.1 pipelining I can
by 1vuio0pswjnm7 3y ago
Think like a www user. Scale down, not up.
For me, the best improvments that have occurred have been sitemaps XML. When paired with HTTP/1.1 pipelining I can peruse and download all pages that I want from a site in a single TCP connection. No CSS, Javascript, images, etc. (Won't find that cruft in Common Crawl either.) This is not "crawling", it's bulk text retrieval. It's starting with a list of URLs from a sitemap XML file and pipelining a series of pages, usually MIME type text/html, in a sequential order, into a compressed archive. Like the KC and the Sunshine Band song, that's the way I like it.
Almost nothing on the www is universal and neither sitemaps nor pipelining are exceptions. Some sites do not even have a robots.txt. (One reason I like HTTP headers they are arguably one of the only universals on the www. All sites have them. Not much variation, either. If "Big Tech" has their way, who knows, that may change, too. HTTP/2 and HTTP/3 seems to give short shrift to HTTP headers. If I'm wrong about that, then that's good to hear.)
Web search engines are opinionated. They are created by people who must make decisions about what to include and what exclude, what to favour and what to disfavour. Obviously, Google's opinion is influenced by advertisers and it hides behind a veil of secrecy as it stacks the deck in its own favour; the company completely abandoned the "search engine in the academic realm" idea Page and Brin wrote about in 1995. It's commercial, but in an insidious, passive-aggressive, "haha no one will notice" sort of way. Marginalia.nu (or wiby.me) might be one of the few non-commercial projects.
Anyway, AFAIK, downloading sitemap XML and then HTML pages in bulk does not give so-called "tech" companies much data to use for advertising or other commercial purposes. It's not like sending one's www search queries to a "search engine".