2 ms·

Ask HN: Best way to scrape web content?

• assafmo 10y ago
  Mostly using linux tools. First of all, always respect the site's terms of use and the server's capacity. It is tempting to just do parallel -P 200 but it can hurt the website and can lead to you being throttled/baned. cURL. Chrome dev tools -> Network -> Copy as cURL. also very easy to customize requests. sessions can be done using --cookie and --cookie-jar. Using tor (sudo apt install tor) with --socks5 localhost:9050 or --socks5-hostname localhost:9050 is great for scraping .onion websites and for anonymity. The holy grail for me is to find a JSON REST API for the website I am scraping. lynx --stdin --dump to extract text if there isn't any interesting stuff inside the html/js (man lynx! it has some great options) awk to extract data. very easy to extract data from tables! to extract data from html I usually use awk -F '[<>]' or awk -F '"'... depends on where the data is in the html. (also sometimes egrep -o to extract with a regex) seq 1 10 | parallel curl https://banana.papaya.com/?page={} https://banana.papaya.com/?page={} is great for paging. jq to process JSONs. I usually output to csv with @csv, because then I can use sqlite for quick queries or any other DB that has an easy "import from csv" option (I like using CouchDB with couchimport). Dynamic pages - You can always get the content using the same request the that site's javascript did. I never find myself needing to execute the page's javascript. EDIT: typos, grammer
  • tmaly 10y ago
    Thanks for the Chrome dev tools tip, I did not even know they had that. lynx brings back memories when all I had was a green screen vt100 terminal in the university library.
• zoobab 10y ago
  I use python selenium with Firefox in headless mode. It can render or not the javascript.
  • tmaly 10y ago
    Do you have an example of any blog posts on how to get it setup?
    • zoobab 9y ago
      I will try to post something.
• btschaegg 10y ago
  For simple, scripted, one-off tasks, I've had good experiences with Beautiful Soup for Python[1]. If it's necessary that the javascript actually gets executed though, you're more likely to need something like headless chrome (recently on HN [2]) or PhantomJS [3], but I can't speak to how easy they are to use, though. [1]: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ https://www.crummy.com/software/BeautifulSoup/bs4/doc/ [2]: https://news.ycombinator.com/item?id=14101233 https://news.ycombinator.com/item?id=14101233 https://news.ycombinator.com/item?id=14239194 https://news.ycombinator.com/item?id=14239194 [3]: http://phantomjs.org/ http://phantomjs.org/
  • tmaly 10y ago
    Thanks, I would love to hear more about the headless chrome if someone can chime in.
• iamsvera 10y ago
  If you know which requests you need to do in order to get the info you want instead of waiting for the headless browser to execute the javascript code that renders the needed data, it is pretty simple, you just need an http client and something to parse the content, depending if is html I'd recommend using cheerio in NodeJS to parse and traverse the content with a jQuery like API. If you must scrape the content with a headless browser, I'd choose Selenium. https://github.com/sourcegraph/go-selenium https://github.com/sourcegraph/go-selenium
  • iamsvera 10y ago
    There is a really cool and easy to use library for NodeJS, but it doesn't use a headless browser :( it just makes the requests and parses the content. https://github.com/IonicaBizau/scrape-it https://github.com/IonicaBizau/scrape-it
  • tmaly 10y ago
    Thanks, have you used go-selenium and what did it take to get up and going with it?