7 ms·
FWIW, this site is fast, works perfectly with a text-only browser, e.g., links (1.4MB static binary), elinks, or without a (bloated?) browser, e.g., retrieve al
by 1vuio0pswjnm7 2mo ago
FWIW, this site is fast, works perfectly with a text-only browser, e.g., links (1.4MB static binary), elinks, or without a (bloated?) browser, e.g., retrieve all the pages, catenate and read as a single HTML file or convert it to, e.g., SQL, CSV or a plain text file
All the /p/ URLs are in the sitemap, all the site's pages can be retrieved over a single TCP connection
https://debloat.dev/sitemap.xml https://debloat.dev/sitemap.xml
This, i.e., retrieving all 200 /p/ URLs over a single TCP connection (using HTTP/1.1 pipelining), results in a 1.9MB HTML file comprising all the /p/ pages, including response headers. NB. This isn't "crawling". All URLs (cf. a set of "seed" URLs) are known before the connection is made
No cookies, no Javascript
Other than CSS, no bloat
NB. A 1.9MB sitemap would not be large. For example, here is a 2.9MB sitemap from AP
https://apnews.com/ap-sitemap-202607.xml https://apnews.com/ap-sitemap-202607.xml
- eterm 2mo agoI about to splutter that 1.9MB is a huge sitemap, but it's 12kB. Do you mean crawling the entire sitemap and concatenating the results is 1.9MB?
- varun_ch 2mo agoAlso, the website seems to be largely ‘vibe coded’, at least given by the written copy and the style. If you ask Claude to use 1990s/2000s web design and HTML it’ll do something like this.
- 1vuio0pswjnm7 2mo agoThe sitemap "standard" was not introduced until 2005 Would a "1990's style" website offer a sitemap.xml file Also, this site, like most today, uses "slugs" for its URLs. In the 1990's this was less common; URLs were often "uncool": https://www.w3.org/Provider/Style/URI.html https://www.w3.org/Provider/Style/URI.html NB. I'm not commenting on the website's content or how it was generated/sourced