6 ms·

Basic Web Scraping with Emacs

• jhabdas 8y ago
  DOM crawling can be a challenge without XPATH. Nice work. The flow looks smooth. Any chance of an el-get?
• nickdrozd 8y ago
  The idea here is that 1) Emacs has flexible means of automation (like keyboard macros), and 2) it also has a web browser, so 3) Emacs can be used for browser automation. The conclusion seems trivial in retrospect, but I hadn't seen it discussed anywhere, so I thought the techniques involved would be worth recording.
  • gkya 8y ago
    Thanks a lot! I use Emacs and have some scraping projects (mainly converting some websites' announcements into RSS feeds), but have never thought of this either.
  • stevenwoo 8y ago
    Just finding about eww was fun for me, I could probably tune browser settings/plugins to approximate it but it makes web browsing on sites I just want to read extremely quick to load no matter how much extraneous baggage is on the web site.
• paulcarroty 8y ago
  Guess dynamical websites aren't supported?
  • c256 8y ago
    JavaScript isn’t run, so some dynamic web sites are supported and others (common today) aren’t.
• stevenwoo 8y ago
  Had to search for how to use eww since I have never used it before now.