4 ms·
The frictionlessdata landing page has very generalized verbiage so here's my technical summary of it... The main idea for "container" or "package" hinges on a
by jasode 9y ago
The frictionlessdata landing page has very generalized verbiage so here's my technical summary of it...
The main idea for "container" or "package" hinges on a file called "datapackage.json"[1].
An analogy would be the "sfv" files like "checksums.sfv" for verifying the integrity of files. Since so many people use "sfv" as a defacto standard, many programs exist to scan it and verify the associated files. Another analogy would be DTD for XML files.
Similarly, if everybody could converge on the file "datapackage.json" as a metadata & schema description standard, a useful ecosystem of utilities and libraries for processing data would take advantage of it.
One example library would be: https://github.com/frictionlessdata/datapackage-py https://github.com/frictionlessdata/datapackage-py
(In the Python source code for "package.py"[2], Ctrl+F search for "datapackage.json" to see how it looks for that particular file.)
With a data wrangling API like that, one could then do joins on csv files directly[3] and write the results to another csv file with the associated "datapackage.json".
Instead of passing "dumb" csv or raw json files around, add a little "intelligence" to the dataset by way of "datapackage.json" so tools can parse the schema and process csv/json at a higher abstraction level. That leads to more "effortless" and "frictionless" data interoperability.
What I can't tell so far is if "datapackage.json" already has momentum of adoption across many communities such as Julia, Tensorflow, Hadoop, etc. and we need to get on the bandwagon -- or -- adoption is still in its infancy and there are other competing data "container/package" specifications to look at.
[1] http://frictionlessdata.io/guides/data-package/ http://frictionlessdata.io/guides/data-package/
[2] https://github.com/frictionlessdata/datapackage-py/blob/master/datapackage/package.py https://github.com/frictionlessdata/datapackage-py/blob/mast...
[3] http://frictionlessdata.io/guides/joining-tabular-data-in-python/ http://frictionlessdata.io/guides/joining-tabular-data-in-py...
- _pwalsh 9y agoHi, (I work on the Frictionless Data specifications and tooling at Open Knowledge International.) Thanks. We are working on the website at present [1], and we are trying to manage a balance of targeting technical and non-technical users, which is hard to get right. About momentum - I can address that. We have seen significant momentum in the last 2 years, around open data / government transparency / civic tech ( our natural environment - see https://okfn.org https://okfn.org for details ), around scientific / academic research via our work enabled by a grant from Sloan [2]( see http://frictionlessdata.io/case-studies/ http://frictionlessdata.io/case-studies/ for a small selection, more reports coming ), and in general around data wrangling and data science efforts (including integration of Table Schema [3] with Pandas [4]). In terms of big data / machine learning - we have not actively worked in that space to present. In terms of Julia, and other languages, we have a Julia library in development via our Tool Fund [5], and this will add to implementations [6] in PHP, Java, R, Clojure which are already underway via the Tool Fund, and accompany the Python, Javascript and Ruby implementations that we maintain directly at Open Knowledge International. [1]: https://github.com/frictionlessdata/frictionlessdata.io/issues/129 https://github.com/frictionlessdata/frictionlessdata.io/issu... [2]: https://sloan.org https://sloan.org [3]: http://specs.frictionlessdata.io/table-schema/ http://specs.frictionlessdata.io/table-schema/ [4]: https://pandas-docs.github.io/pandas-docs-travis/generated/pandas.io.json.build_table_schema.html https://pandas-docs.github.io/pandas-docs-travis/generated/p... [5]: http://toolfund.frictionlessdata.io http://toolfund.frictionlessdata.io [6]: https://github.com/frictionlessdata https://github.com/frictionlessdata