3 ms·
This is about the scraping tools that are used to train LLMs, not the scrapers used by live LLMs.
by seszett 1y ago
This is about the scraping tools that are used to train LLMs, not the scrapers used by live LLMs.
- sltkr 1y agoEven the ones used live, why couldn't/wouldn't they run some type of headless browser with Javascript enabled, rather than just making an HTTP-request and looking at the HTML code?
- seszett 1y agoThat's the point, the scraping that happens live can run through a full browser, performance is not an important issue there The scraping that's taking place to train the models is in a completely different scale and (probably) cannot really bear the added cost of PoW on each page they scrape. That's the scraping that is targeted by these tools.