5 ms·

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented da

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice. * Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice. * Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it. Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-ideal-world https://github.com/HeinrichHartmann/data-sharing#in-the-idea... Approaches I have seen so far in the wild: * figshare.com -- Addresses Hosting and Presentation. * https://quiltdata.com/ https://quiltdata.com/ -- (!) looks great. Still exploring. * github.com -- works fine for small datasets (<1GB) * packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance. This seems to be a wide open problem to me.

rgardaphe 8y ago
Totally agree! At Qri (https://qri.io https://qri.io) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!
- sixdimensional 8y ago
  This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult. For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was. Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!
  - rgardaphe 8y ago
    Thanks! you can follow us on twitter where we announce most updates: @qri_io
- heinrichhartman 8y ago
  That looks interesting. IPFS backed -- nice. A web interface would be nice so that I don't have to install a tool to browse the content. Going to give it a shot non-theless.
  - heinrichhartman 8y ago
    Hmm: "Qri backend is unavailable". Going to come back tomorrow.
    - ramfox 8y ago
      Hey! thanks for checking us out. If you are still having trouble, please head over to our github (http://www.github.com/qri-io/frontend http://www.github.com/qri-io/frontend) and I'd be happy to help out. You would be doing us a huge solid, working through these use cases irl is beyond helpful.
sixdimensional 8y ago
I agree and I have an idea for one possible solution that I've wanted to implement for years. It could be a business but, I think maybe it would be better for the world if it was open source. I just haven't had the time or support to do it, as it would need full-time effort even to get it started. It's one of those... I know I should just do it kind of things, even just to get it out there, but I haven't found the inertia. Seeing things like dat, quiltdata, public data sets, etc. made me think what I wanted to do was unnecessary, but I also agree with your comment. I think a core problem is data democracy / control / politics of data. Too much we still act silo'ed instead of benefiting from massive data sharing, for a multitude of reasons (especially but not limited to $$$).
rxdazn 8y ago
Open Data portals powered by OpenDataSoft let you see/sort/filter/visualize data in your browser. You can access the data via API (including sorting/filtering) or download static files (CSV/JSON/XLS etc.). I haven't seen many other platforms offer the same kind of functionality. e.g This one is a dataset of CCTVs across Leicester. You can easily see all of the columns, sort data around, display a chart of camera types, see their location on a map etc. https://data.opendatasoft.com/explore/dataset/cctv-cameras%40leicester/ https://data.opendatasoft.com/explore/dataset/cctv-cameras%4... Screenshots: https://postimg.cc/gallery/j5e5thv2/ https://postimg.cc/gallery/j5e5thv2/ A few portals powered by them: - Paris https://opendata.paris.fr/explore/ https://opendata.paris.fr/explore/ - Mannheim https://mannheim.opendatasoft.com/page/home/ https://mannheim.opendatasoft.com/page/home/ - Durham NC https://opendurham.nc.gov/pages/home/ https://opendurham.nc.gov/pages/home/ disclaimer: I was an intern there 4 years ago
rufuspollock 8y ago
Yes, state is still mediocre. Back in 2011 i did Chaos Computer Congress talk on "apt-get for Debian of Data". This itself came out of building the original data portal and open data catalog/hosting site in 2006 https://datahub.io/ https://datahub.io/ (originally ckan.net). The ideas there are now getting realised in Frictionless Data https://frictionlessdata.io/ https://frictionlessdata.io/ This an initiative providing a simple way of "packaging" data like software plus an ecosystem of tools including a package manager etc - https://frictionlessdata.io/data-packages/ https://frictionlessdata.io/data-packages/. Aims to be minimal, easy to adopt etc (e.g. based on CSV). It has got significant traction with integration and adoption into Pandas, OpenRefine etc. https://datahub.io/ https://datahub.io/ itself is entirely rebuilt around Data Packages and includes a package manager tool "data". If you're interested to talk more please come chat on http://gitter.im/datahubio/chat http://gitter.im/datahubio/chat
dandare 8y ago
Shameless plug. Please give me feedback. https://us.wikibudgets.org/w/united-states-budget-2016 https://us.wikibudgets.org/w/united-states-budget-2016 I would like to create laymen oriented central repository for all public spending data of the world. * Hosting problems - I make my own copy of the data. * Format problems - clearing and formatting data from different sources is a real pain. Once it is on my website, I offer CSV download or COPY/PASTE tabulated data. * Update problems - no versioning or public API yet. * Provenance problems - there is a link to the source of the data. * Presentation problems - tailored to displaying budgets. Not cross-browser or full mobile support yet.

rgardaphe 8y ago

Totally agree! At Qri (https://qri.io https://qri.io) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!

sixdimensional 8y ago
This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult. For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was. Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!
- rgardaphe 8y ago
  Thanks! you can follow us on twitter where we announce most updates: @qri_io
heinrichhartman 8y ago
That looks interesting. IPFS backed -- nice. A web interface would be nice so that I don't have to install a tool to browse the content. Going to give it a shot non-theless.
- heinrichhartman 8y ago
  Hmm: "Qri backend is unavailable". Going to come back tomorrow.
  - ramfox 8y ago
    Hey! thanks for checking us out. If you are still having trouble, please head over to our github (http://www.github.com/qri-io/frontend http://www.github.com/qri-io/frontend) and I'd be happy to help out. You would be doing us a huge solid, working through these use cases irl is beyond helpful.

sixdimensional 8y ago

I agree and I have an idea for one possible solution that I've wanted to implement for years. It could be a business but, I think maybe it would be better for the world if it was open source. I just haven't had the time or support to do it, as it would need full-time effort even to get it started. It's one of those... I know I should just do it kind of things, even just to get it out there, but I haven't found the inertia. Seeing things like dat, quiltdata, public data sets, etc. made me think what I wanted to do was unnecessary, but I also agree with your comment. I think a core problem is data democracy / control / politics of data. Too much we still act silo'ed instead of benefiting from massive data sharing, for a multitude of reasons (especially but not limited to $$$).

rxdazn 8y ago

Open Data portals powered by OpenDataSoft let you see/sort/filter/visualize data in your browser. You can access the data via API (including sorting/filtering) or download static files (CSV/JSON/XLS etc.). I haven't seen many other platforms offer the same kind of functionality. e.g This one is a dataset of CCTVs across Leicester. You can easily see all of the columns, sort data around, display a chart of camera types, see their location on a map etc. https://data.opendatasoft.com/explore/dataset/cctv-cameras%40leicester/ https://data.opendatasoft.com/explore/dataset/cctv-cameras%4... Screenshots: https://postimg.cc/gallery/j5e5thv2/ https://postimg.cc/gallery/j5e5thv2/ A few portals powered by them: - Paris https://opendata.paris.fr/explore/ https://opendata.paris.fr/explore/ - Mannheim https://mannheim.opendatasoft.com/page/home/ https://mannheim.opendatasoft.com/page/home/ - Durham NC https://opendurham.nc.gov/pages/home/ https://opendurham.nc.gov/pages/home/ disclaimer: I was an intern there 4 years ago

rufuspollock 8y ago

Yes, state is still mediocre. Back in 2011 i did Chaos Computer Congress talk on "apt-get for Debian of Data". This itself came out of building the original data portal and open data catalog/hosting site in 2006 https://datahub.io/ https://datahub.io/ (originally ckan.net). The ideas there are now getting realised in Frictionless Data https://frictionlessdata.io/ https://frictionlessdata.io/ This an initiative providing a simple way of "packaging" data like software plus an ecosystem of tools including a package manager etc - https://frictionlessdata.io/data-packages/ https://frictionlessdata.io/data-packages/. Aims to be minimal, easy to adopt etc (e.g. based on CSV). It has got significant traction with integration and adoption into Pandas, OpenRefine etc. https://datahub.io/ https://datahub.io/ itself is entirely rebuilt around Data Packages and includes a package manager tool "data". If you're interested to talk more please come chat on http://gitter.im/datahubio/chat http://gitter.im/datahubio/chat

dandare 8y ago

Shameless plug. Please give me feedback. https://us.wikibudgets.org/w/united-states-budget-2016 https://us.wikibudgets.org/w/united-states-budget-2016 I would like to create laymen oriented central repository for all public spending data of the world. * Hosting problems - I make my own copy of the data. * Format problems - clearing and formatting data from different sources is a real pain. Once it is on my website, I offer CSV download or COPY/PASTE tabulated data. * Update problems - no versioning or public API yet. * Provenance problems - there is a link to the source of the data. * Presentation problems - tailored to displaying budgets. Not cross-browser or full mobile support yet.