4 ms·
If you're the kind of person who wants "open data" (read as broadly as you like) and could get it in snapshots direct from the source without having to scrape,
by afandian 4y ago
If you're the kind of person who wants "open data" (read as broadly as you like) and could get it in snapshots direct from the source without having to scrape, what would your ideal format be?
I know it's a very open ended question.
- mdaniel 4y agoAlternatively, think how much traffic a site could save itself if it allowed subscribing to webhooks on new content, or having a functional sitemap.xml to say nothing of the end-user experience of having fresh content I would speculate (based on anecdata) that a lot of the actual load placed upon sites is from the discovery phase -- what pages are there, and have any of them changed -- not so much "hit this one endpoint and unpack its data"
- tasn 4y agoIt's an interesting idea to have a sort of a "what's up?" endpoint that is just basic text and tells you what's fresh. At least for static websites, it's quite easy to do.
- dwrodri 4y agoThe way that PushShift offers huge amounts of reddit data in compressed JSON is very convenient. Checkout files.pushshift.io
- jshen 4y agoProbably RDF serialized as hextuples https://github.com/ontola/hextuples https://github.com/ontola/hextuples
- afandian 4y agoLooks interesting. From that page I couldn't see what 'graph' field relates to. Is it the identifier for a distinct named graph? It was blank in the examples. Do you use it? What for?
- jshen 4y agoIt’s not well defined. I try to use it to represent where the data came from, I.e. dbpedia
- PigiVinci83 4y agoThanks for the question, i can speak for what we've encountered in these years of web scraping and nothing beats API and JSON, but i'm sure there are formats even more friendly to read.