3 ms·
A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use w
by softwaredoug 5y ago
A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use web scraping. This in turn puts load and complexity on the user facing web app that must now tell scraper from real users.
- fragmede 5y agoBut if there's an API, then the overall load is the same, no? Or to put it another way, naively, having api.example.com and realpeople.example.com separated out into separate sandboxes seems reasonable, but due to the aforementioned problem, its not. But then it also turns out to be the wrong axis for this anyway, and you need your monitoring to work for you.
- kingcharles 5y agoNo, the load isn't the same because the web page might be a multi-megabyte monster piece of badly-coded HTML that returns only say 10 out of 1,000,000 results and needs to be paged through, where the API might return all the million results in a nice JSON chunk.
- no_time 5y agoUsing the API almost always has more "strings attached". Like you have to register and get an API token or something. Or even pay. If you want people to use your API, don't make it less convinient than scraping the page.