3 ms·
This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie prog
by danso 10y ago
This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that's increasingly less common the case. So now you have to teach how to debug via the console and network panel, on top of basic HTTP concepts (such as query parameters).
(Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that's another issue).
I've been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX/app-like sites, if the trend at the federal level means anything.
That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML.
But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It's been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back
[0] https://github.com/wgetsnaps https://github.com/wgetsnaps
[1] https://github.com/wgetsnaps/tdcj-state-tx-us--death_row https://github.com/wgetsnaps/tdcj-state-tx-us--death_row
[2] https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92 https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92
- niftich 10y agoIt's unfortunate that nearly every webpage these days is a Javascript State Machine which you have to execute in a sandbox and inspect its internal state to get stuff out of. On a blog post by Paul Kinlan ('Open Web Advocate' at Google and Chromium) [1], I lamented that we ended up here instead of the semantic web because the semantic web was hard to execute. Instead, every web page is a black-box, only navigable by an intelligent and/or sufficiently persuadable human. But this is also why I don't buy ethical arguments against scraping. Sure, legally any company can unilaterally set any TOS prohibition against behavior they don't want, and these terms may be tested in court. But navigating a page in an automated manner that's designed to resemble interactions of humans (ie. through Selenium) is in my opinion ethical, because it merely time-shifts a user's activity. [1] https://news.ycombinator.com/item?id=12206846 https://news.ycombinator.com/item?id=12206846