4 ms·
This. A lot of modern sites can be really easy to scrape. Lots of machine-readable data. APIs (for SPAs), OpenGraph/LD+JSON data in <head>, and data- attribute
by deanishe 3y ago
This. A lot of modern sites can be really easy to scrape. Lots of machine-readable data.
APIs (for SPAs), OpenGraph/LD+JSON data in <head>, and data- attributes with proper data in them (e.g. a timestamp vs "just now" in the text for the human).
Scraping is a lot easier than it used to be.
- zffr 3y agoAdding on to this, if an app uses client-side hydration (ex Next apps) sometimes you can find a big JSON object in the HTML with all the page data. In these cases you can usually write some custom code to extract and parse this JSON object. Sometimes the JSON is embedded in some JavaScript code so you need to use a little regex to extract it.