3 ms·

Contrary to popular belief, Ajax is actually one of the best things to have happened for hooking up to Web Apps and scraping. You used to have to walk through t

Contrary to popular belief, Ajax is actually one of the best things to have happened for hooking up to Web Apps and scraping. You used to have to walk through the obscure rendered html, now you mostly get to access the raw data in json format.

jtmarmon 10y ago
is this really a contrarian opinion? i feel like this is a pretty obvious fact that it's easier to read network requests to scrape a website than parsing html
- alanwatts 10y ago
  Is there a specific tool you would recommend for doing this?
  - Jonovono 10y ago
    To read network requests? Charles proxy is great!
  - slig 10y ago
    Chrome Dev Tools is pretty awesome. You can right-click a request and get an 'curl' request with cookies and everything that can be replayed on a terminal.
- thomasahle 10y ago
  I'm not sure why, but many scraping tools handle javascript websites by emulating clicks and running the javascript.
  - jandrese 10y ago
    I think the idea is that clicking on the button is less likely to change suddenly than whatever protocol they're running on top of AJAX. In my experience internal layout changes seem to happen way more often than changes to the AJAX handlers.