3 ms·
So I have seen that indeed! But also, some more modern sites with JSON API endpoints will have extremely bespoke session/auth/state management systems that mak
by btown 3y ago
So I have seen that indeed!
But also, some more modern sites with JSON API endpoints will have extremely bespoke session/auth/state management systems that make it difficult to create a request payload that will work without calculations done deep in the bowels of their client-side JS code. It can be much easier, if slower and more costly, to mimic a browser and listen to the equivalent of the Network tab, than to find out how to create valid payloads directly for the API endpoints.
- josephg 3y agoIf you can see the request in the network tab, you can just right click - copy as curl and then replay the request from the command line and noodle with the request parameters that way. Works great!
- kbenson 3y agoYes. Sometimes you can see that there's a static (per session) header they add to each request, and all you have to do is find and record that header value (such as shimming addRequestHeader) and append it to your own requests from that context...
- ricardo81 3y agoHonestly, from prior experience any scraping requirements that require browser implementation tend to be due to captchas and anti-scraping measures, nothing to do with the data layout. It's either in the DOM or in one or two other payloads. Isn't this sort-of-why people hide themselves behind Cloudflare, to remove the lowest common denominators of scraping.