4 ms·
> is more ethical How do you deal with pages that use JS to load their content (e.g. SERPs) and restrict those endpoints to be called from within that page? I
by woodpanel 4y ago
> is more ethical
How do you deal with pages that use JS to load their content (e.g. SERPs) and restrict those endpoints to be called from within that page?
I'm lucky if I can use cheerio to just traverse the DOM on a given page, but increasingly I have to render the page and that "scales" as well, at least in terms of maintainability since I can more or less use the same API to traverse the (then JS-modified) DOM
- deleted 4y ago[deleted]
- 10000truths 4y agoYou can observe the network requests that the JS makes under the Network tab of the Developer Tools console. The restrictions you mention can be bypassed by setting the Origin and Referer HTTP headers to whatever satisfies the server.