3 ms·
The page requires JS to load its content - user agents without JS support just get a blank page. I'm not sure if the bots that scrape data to train LLMs are ca
by everyos_ 7mo ago
The page requires JS to load its content - user agents without JS support just get a blank page.
I'm not sure if the bots that scrape data to train LLMs are capable of loading that type of page, or if they only work on pages that have the content inside the HTML itself?
- replygirl 7mo agoany serious scraping service these days will fail over to a headless browser when it fetches an asset referencing a js bundle that isn't verifiably a vendor script
- bstrama 7mo agoI'm aware and will implement SSR soon ;)
- m3047 7mo agoIt's entirely possible they simply ingest the JS as-is.
- aDyslecticCrow 7mo agoNot using JavaScript would also make the crawler fail on squarespace and wix website builders. The age where the web was usable at all without JavaScript is long gone. No scraper would get much scraping done without JavaScript these days.
- cachius 7mo agoYou mean by embedding? How can an external site fail on squarespace and wix website builders?
- tardedmeme 7mo agoA crawler would fail on all Squarespace and Wix sites if they all require JavaScript.
- cachius 7mo agoFound it https://halupedia.com/javascript-requirement-on-squarespace-and-wix-website-builders https://halupedia.com/javascript-requirement-on-squarespace-...