3 ms·

I don't really know why human readable is important here, but there already exist open data formats that have good support for the things that matter: large dat

by jpallen 13y ago

I don't really know why human readable is important here, but there already exist open data formats that have good support for the things that matter: large data sets, compression and support for multiple platforms and languages. I'm not very knowledgeable about any, but some I am aware of: http://www.unidata.ucar.edu/software/netcdf/ http://www.unidata.ucar.edu/software/netcdf/ http://cdf.gsfc.nasa.gov/ http://cdf.gsfc.nasa.gov/

toufka 13y ago
Entirely different concept. I want simple data to be easily distributable and interpretable. Make a format that can universally read/write the data in these very basic graphs: http://download.cell.com/images/journalimages/2211-1247/PIIS2211124713001964.gr3.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS2211124713001721.gr2.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS2211124712004299.gr4.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... Most science (that really needs hackers' help) is not the big sexy datasets, but the really basic HTML5/CSS viewable graphs otherwise done in Excel. But it must be done in a scientifically rigorous way (retain units and labels, be able to manipulate those units, and be able to show/hide data between different datasets). Think a way to download and view a melting point curve dataset without ANY user configuration. Or an enzyme activity curve. Or a drug dosage response curve. VERY basic data, but it must be able to maintain and interpret a label and a unit in addition to the number itself.
- jpallen 13y ago
  I don't necessarily disagree with you, I just always prefer to sit more on the side of 'don't reinvent the wheel if you don't have to'. I don't think creating an n-th competing format is the answer here. In the examples in your final paragraph, surely what is needed there is a new tool, not a new data format? I think the larger 'problem' (if it really is a problem, and not just hackers wanting to believe they know the One True Way) is that there is no such thing as the typical 'scientist'. Every small sub-discipline has its own needs, its own standards that it has settled on and its own toolchain. There is definitely room for improvement within every one of these small areas, but there's no global problem that needs solving.
  - toufka 13y ago
    Part of this is very true. Part of it is that science has already done this - 200 years ago with the development of scientific units. We already have a great universal format - it is just not digital in nature and doesn't exist as a 'file format'.

by jpallen 13y ago

toufka 13y ago

Entirely different concept. I want simple data to be easily distributable and interpretable. Make a format that can universally read/write the data in these very basic graphs: http://download.cell.com/images/journalimages/2211-1247/PIIS2211124713001964.gr3.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS2211124713001721.gr2.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... http://download.cell.com/images/journalimages/2211-1247/PIIS2211124712004299.gr4.lrg.jpg http://download.cell.com/images/journalimages/2211-1247/PIIS... Most science (that really needs hackers' help) is not the big sexy datasets, but the really basic HTML5/CSS viewable graphs otherwise done in Excel. But it must be done in a scientifically rigorous way (retain units and labels, be able to manipulate those units, and be able to show/hide data between different datasets). Think a way to download and view a melting point curve dataset without ANY user configuration. Or an enzyme activity curve. Or a drug dosage response curve. VERY basic data, but it must be able to maintain and interpret a label and a unit in addition to the number itself.

jpallen 13y ago
I don't necessarily disagree with you, I just always prefer to sit more on the side of 'don't reinvent the wheel if you don't have to'. I don't think creating an n-th competing format is the answer here. In the examples in your final paragraph, surely what is needed there is a new tool, not a new data format? I think the larger 'problem' (if it really is a problem, and not just hackers wanting to believe they know the One True Way) is that there is no such thing as the typical 'scientist'. Every small sub-discipline has its own needs, its own standards that it has settled on and its own toolchain. There is definitely room for improvement within every one of these small areas, but there's no global problem that needs solving.
- toufka 13y ago
  Part of this is very true. Part of it is that science has already done this - 200 years ago with the development of scientific units. We already have a great universal format - it is just not digital in nature and doesn't exist as a 'file format'.