4 ms·
The page requires JS to load its content - user agents without JS support just get a blank page. I'm not sure if the bots that scrape data to train LLMs are ca
by everyos_ 5mo ago
The page requires JS to load its content - user agents without JS support just get a blank page.
I'm not sure if the bots that scrape data to train LLMs are capable of loading that type of page, or if they only work on pages that have the content inside the HTML itself?
- replygirl 5mo agoany serious scraping service these days will fail over to a headless browser when it fetches an asset referencing a js bundle that isn't verifiably a vendor script
- bstrama 5mo agoI'm aware and will implement SSR soon ;)
- m3047 5mo agoIt's entirely possible they simply ingest the JS as-is.
- aDyslecticCrow 5mo agoNot using JavaScript would also make the crawler fail on squarespace and wix website builders. The age where the web was usable at all without JavaScript is long gone. No scraper would get much scraping done without JavaScript these days.
- cachius 5mo agoYou mean by embedding? How can an external site fail on squarespace and wix website builders?
- tardedmeme 5mo agoA crawler would fail on all Squarespace and Wix sites if they all require JavaScript.
- cachius 5mo agoFound it https://halupedia.com/javascript-requirement-on-squarespace-and-wix-website-builders https://halupedia.com/javascript-requirement-on-squarespace-...