3 ms·
Most scrapers don't run browsers. Making the scrapers run real browsers changes the economics of scraping and makes it less cost-effective. I am working on mak
by xena 1y ago
Most scrapers don't run browsers. Making the scrapers run real browsers changes the economics of scraping and makes it less cost-effective.
I am working on making it allow more traffic by default and then applying challenges based on request pressure or other factors like system load. I also need to finish the WebAssembly PR and a few other important things.
It's a work in progress, but it's used by the United Nations so it can't be that bad :)
- JohnFen 1y ago> Most scrapers don't run browsers. The specific risk I was talking about was that if OpenAI buys Chrome, they could (and I think it's likely they would) use the contents of whatever pages the users browse to as training for their models. Basically, turning the browser into a disguised crawler that would be immune from the usual anti-crawling methods, including putting up a login page.
- ArinaS 1y agoSo literally spyware. Don't get me wrong, current Chrome is also spyware, but sending every page you visit to a third-party is just a whole new level.