3 ms·

No mention of Javascript. All the pages that I would consider scraping are constructed, at least in part, client side by Javascript. If that Javascript is not

No mention of Javascript. All the pages that I would consider scraping are constructed, at least in part, client side by Javascript. If that Javascript is not executed then there is nothing interesting to scrape.

traverseda 5y ago
That javascript is presumably fed by APIs, and often capturing the content of those APIs is better than capturing the rendered view. That can be a lot of work though, use selenium or the more modern playwright to run entire web pages in a remote-controlled browser.
maltz 5y ago
If pages are constructed client-side, the content you are looking for is either hardcoded as JSON in the HTML or loaded via XHR request. Scrape that.
rstuart4133 5y ago
That's because javascript isn't relevant. The _only_ way the browser can interact with the server is via http requests. That's the level the scraper operates at - imitating the http requests the browser does. In particular, it doesn't matter why the browser did those http requests. It could be because the user submitted a form, or clicked a link, or javascript did some AJAX request, it's done by a web worker or browser plugin, or god help us calls some function some Active-X component. Provided the scraper emulates http request perfectly, there is no way server can tell if the request came from the component it expects or a scraper. It is both a benefit and a curse. It's a benefit because all the complexity of javascript libraries, DOM's and what not goes away. For example, back in the day I've scraped the satellite imagery from maps.google.com. Maps is a giant horridly complex javascript application - you really want avoid understanding how it does what it does. The http requests it makes on the other hand are pretty simple. However Google didn't want you scrapping it, so they included authentication in there. Authentication always boils down to taking some data they sent in a previous request, mangling it with javascript then sending it back as a cookie or hidden field in a form. You have to replicate that mangling perfectly, which involves reading and understand the minimised javascript. That's the curse. Such reverse engineering can take a while, but it's mercifully rare. The payoff is speed, and reduced fragility. The speed comes arises because most of the crap a browser downloads is only useful to human eyes, and the scraper doesn't have to download it. Fragility is reduced because GUI's, even web GUI's and especially javascript laden SPA's often want mouse clicks and keystrokes in a certain order, and while particular parts of the screen have focus. For some reason web designers love tweaking their UI's which breaks that order. The data they send back with their forms and AJAX requests is far more stable.

traverseda 5y ago

That javascript is presumably fed by APIs, and often capturing the content of those APIs is better than capturing the rendered view. That can be a lot of work though, use selenium or the more modern playwright to run entire web pages in a remote-controlled browser.

maltz 5y ago

If pages are constructed client-side, the content you are looking for is either hardcoded as JSON in the HTML or loaded via XHR request. Scrape that.

rstuart4133 5y ago

That's because javascript isn't relevant. The _only_ way the browser can interact with the server is via http requests. That's the level the scraper operates at - imitating the http requests the browser does. In particular, it doesn't matter why the browser did those http requests. It could be because the user submitted a form, or clicked a link, or javascript did some AJAX request, it's done by a web worker or browser plugin, or god help us calls some function some Active-X component. Provided the scraper emulates http request perfectly, there is no way server can tell if the request came from the component it expects or a scraper. It is both a benefit and a curse. It's a benefit because all the complexity of javascript libraries, DOM's and what not goes away. For example, back in the day I've scraped the satellite imagery from maps.google.com. Maps is a giant horridly complex javascript application - you really want avoid understanding how it does what it does. The http requests it makes on the other hand are pretty simple. However Google didn't want you scrapping it, so they included authentication in there. Authentication always boils down to taking some data they sent in a previous request, mangling it with javascript then sending it back as a cookie or hidden field in a form. You have to replicate that mangling perfectly, which involves reading and understand the minimised javascript. That's the curse. Such reverse engineering can take a while, but it's mercifully rare. The payoff is speed, and reduced fragility. The speed comes arises because most of the crap a browser downloads is only useful to human eyes, and the scraper doesn't have to download it. Fragility is reduced because GUI's, even web GUI's and especially javascript laden SPA's often want mouse clicks and keystrokes in a certain order, and while particular parts of the screen have focus. For some reason web designers love tweaking their UI's which breaks that order. The data they send back with their forms and AJAX requests is far more stable.