4 ms·
Yeah, that was what I did when I didn't know anything about programming, too. The next step is to learn why web scraping is discouraged, and how to use public
by code_duck 16y ago
Yeah, that was what I did when I didn't know anything about programming, too.
The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
- naithemilkman 16y agoWhat if the site doesn't have an API? Or what is the API is not user friendly? Case in point: address book scraping employed by many companies bypassing APIS. e.g dropbox
- jacquesm 16y ago> The next step is to learn why web scraping is discouraged, and how to use public APIs instead. Right, just like google uses public api's to access all those sites. Web scraping is responsible for a very large portion of all the traffic on port 80 and the number of sites that have published APIs is extremely small compared to the total.
- code_duck 16y agoI know, it's odd how some people act like all web scraping is a terrible thing, when people have built such large businesses based on it. Companies try very hard to get scraped by Google, and don't like being scraped by other random entities so much - I suppose it depends on your intent. Poorly written spiders are a problem for system admins. If a resource does have a public API, some very positive things about using that instead are that not only is it less resource intensive for them, it is much easier for you to create and maintain. If you're relying on scraping to get specific info from a site, it's a huge Odin when changes to their HTML break your code.
- templaedhel 16y agoYep, I access HN via a powerful rest API.