3 ms·
Ask HN: Why is there no adopted standard for web scraping?
I feel many sites benefit from scraping, or are happy to allow it within certain bounds. My question is why have no standards emerged around how data is structured and accessed for web scraping?
I am aware of standards such as Open Graph - is this the closest we have gotten to a standard of machine accessible data on the web?
- friend-monoid 5y agoI few like there has been a lot of attempts at this; the semantic web, html5 semantic tags (article, header, ...), stuff like robots.txt. Developers really like to generalise. Getting everyone to cooperate is the hard part.
- dennisy 5y agoWould you say the issue maybe is that there has not been an easy way to implement the various standards? If we had this automated for web devs the uptake would be much better?
- edent 5y agoSchema.org is probably the closest thing. But, if people want to make their data available, an API is probably the best way.
- dennisy 5y agoAgree, but the API should have a standard - hence my question. Making your site available via an API would be awesome!