4 ms·
At a media company, our web performance monitoring tool started flagging long-running clientside XHR requests, which I couldn't reproduce in a real browser. It
by benregenspan 1y ago
At a media company, our web performance monitoring tool started flagging long-running clientside XHR requests, which I couldn't reproduce in a real browser. It turned out that an analytics vendor was injecting a script which checked if it looked like the client was a bot. If so, they would then essentially use the client as a worker to perform their own third-party API requests (for data like social share counts). So there's definitely some prior art for this kind of thing.
- apitman 1y agoThis is really interesting. One naive thought that immediately came to mind is that bots might be capable of making cross site requests. The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. Not sure that fact will appreciably reduce their scaping abilities though.
- benregenspan 1y ago> The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers I think this is almost already the case now. Services like Cloudflare do a pretty good job of classifying primitive bots and if site operators want to block all (or at least vast majority), they can. The only reliable way through is a real browser. (Which does put a floor on resource needs for scraping)
- dragonwriter 1y ago> The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. I thought bots using (headless) browsers was an existing workaround for a number of existing issues with simpler bots, so this doesn't seem to be a big change.