3 ms·

Why are scrapers and scraping so popular? What is a real use case for it?

• stephengillie 9y ago
  Not seeing ads. Secondarily, there is a lot of data on the internet stored only in HTML pages. For data with multiple sources, HTML is still usually a common format. HTML just has more punctuation and errata to filter out than JSON, XML, or CSV.
  • stesch 9y ago
    So you want the data but you don't want to ask for it?
    • stephengillie 9y ago
      If I come across data I want/need, why should I have to highlight, copy, and read from my clipboard? It's much more elegant to have automation do this.
      • deleted 9y ago
        [deleted]
• troels 9y ago
  Most businesses I've worked for/with had one reason or another to scrape data from various places. Often it's watching competitors, but can also be part of an automation of some process that is already happening manually.
• weberc2 9y ago
  Accessing information that isn't available via an API. For example, much information on HN is only available via scraping. Many banks can't afford to rebuild their backend (probably largely due to the liability/compliance costs) to support an API/client model, but they still want an application--such applications are pretty much required to scrape said banks HTML.
• conradfr 9y ago
  As an example: I listen to the radio a lot and have always wanted to make a website with radio schedules like there is for TV. As this data is not available (at least not in France) I scrap [1] each radio website everyday to get it. [1] Using https://github.com/rchipka/node-osmosis https://github.com/rchipka/node-osmosis for now