2 ms·
I wish websites could just serve the data worth scraping as nice data files, along the content. But it's impossible due to most websites' business model. Those
by nine_k 3y ago
I wish websites could just serve the data worth scraping as nice data files, along the content. But it's impossible due to most websites' business model.
Those who can do it for free, do; see e.g. Wikipedia.
More websites could offer a paid API though; stopping to waste your engineers' time on adding anti-scraping measures, and collecting payments from serious actors could offset the possible losses from copycats.
- dewey 3y ago> More websites could offer a paid API though If you are offering a paid API it becomes a product that's most likely very different from your main product you should be focusing on. So instead of deprecating fields, breaking the format and quickly iterating you'll have to version APIs, add authentication, keep a change log for your paid customers, have customer support for your API customers and make sure you are not doing backwards incompatible changes.
- nine_k 3y agoThis is correct. But if you have a reasonable PMF and stability, and already serve your pages via some API anyway, maybe productizing a portion of that might be worth it.
- isbvhodnvemrwvn 3y agoFunnily enough the site which changed to next.js did, they actually had a lovely offers.json file (~1MB, thousands of offers, just without a long-form HTMLy description) but they were also using it for their transactional needs (meaning that everyone had to load this file before the offers showed up). So they rewrote everything to next.js instead of adding some pagination...
- Alifatisk 3y agoRails has a cool built-in feature where if you add ".json" to the end of the URL, the server will return the same page but in json format with only the data.