4 ms·
You're right that most of the time you don't need to use JavaScript. But look at Google Groups for example - there is an infinite scroll to get all the topics,
by jancurn 11y ago
You're right that most of the time you don't need to use JavaScript.
But look at Google Groups for example - there is an infinite scroll to get all the topics, posts are also loaded dynamically, so you have to wait some time to get them.
In the SFO flights example you have to deal with pagination also using JavaScript.
We wanted to build a powerful tool which can crawl and scrape almost any website out there. It's slower, but you can use bench of our nodes to do it in parallel.
- deleted 11y ago[deleted]
- thecodemonkey 11y agoI agree that the Google Groups example is much simpler when using PhantomJS, but I would argue that it would be an outlier. The SFO flights example is actually heavily over engineered, from quickly glancing over the XHR tab in Chrome Network tools it was pretty obvious that all of the data is actually located in this very nice JSON blob http://www.flysfo.com/flightprocessing/fullFlightData.txt http://www.flysfo.com/flightprocessing/fullFlightData.txt (I assume that the SF Flight Info was just meant as an example for the platform and as such the fact that it's already a JSON blob was just ignored for the sake of the example)