4 ms·

As far as I know this is slowly changing. According to scientists I've met (life science fields) there is an ongoing trend towards demanding raw data and even t

by stinos 9y ago

As far as I know this is slowly changing. According to scientists I've met (life science fields) there is an ongoing trend towards demanding raw data and even the code used to analyze it. Not every journal everywhere yet of course, but the consensus seemed to be that in 5 to 10 years it would become the new norm. Which might cause quite the disruption. Having seen quite the amounts of code written by PHD students I'm 100% sure tere are many, many bugs out there. Possibly leading to faulty results.

c12 9y ago
I think this is a good trend. The opening of raw data AND the code used to analyse it will aid in others being able to find errors sooner.
vanderZwan 9y ago
This is exactly the motivation behind the Loom file format[0] being developed and used in the molecular neurobiology-group I work for[1]. It is an HDF5-based file format for 'omics data, to deal with the ever-growing size of the data sets. I specifically work on the Loom Viewer[1], a SPA that we're trying to design in such a way that it will be really easy and cheap for research groups to host and share these loom files themselves. This would make easy for other groups to ask simple questions about each others data, and in the worst case the raw loom file is always available for download. We're already hosting some of our own published datasets with this viewer, you can check it out here[3]. To lower expectations a bit: the viewer is not trying to be comparable to the big atlases like [4] or [5] (I mean, it's being developed by one dude - me - so by comparison it's a no-budget OSS project). It's much simpler and basic - the idea is that if you use the Loom file format in your pipeline in a sensible manner, the viewer will more or less know what to do with the data. [0] https://github.com/linnarsson-lab/loompy https://github.com/linnarsson-lab/loompy [1] http://linnarssonlab.org/ http://linnarssonlab.org/ [2] https://github.com/linnarsson-lab/loom-viewer https://github.com/linnarsson-lab/loom-viewer [3] http://loom.linnarssonlab.org/ http://loom.linnarssonlab.org/ [4] http://mouse.brain-map.org/agea http://mouse.brain-map.org/agea [5] https://www.proteinatlas.org/tissue https://www.proteinatlas.org/tissue
- reallymental 9y ago
  This is a significant amount of effort, and has some very good touches. Especially the Loom-Viewer, I really liked the ease with which one's workflow can integrate into the data analysis.
  - vanderZwan 9y ago
    Thanks! :) It's always good to hear that we're on the right track with usability. It's one of the things that I'm most worried about: I've been working pretty isolated for the last year and a half, and lack a background in biology or bioinformatics, and did not even have webdev experience when I took on this project (plenty of embarrassing proof of that in the code). Kudos to Sten Linnarsson, the PI of the group and my boss, for taking a gamble and hiring me anyway. > I really liked the ease with which one's workflow can integrate into the data analysis. Just to make this clear: the file format is a "dumb" data storage, and the viewer a "dumb" plotter of that data. To do more in-depth analysis requires loading the file in python, R, or anything else that might support the files in the future. The idea is to then store the results of this as attributes in the file. For example, the tSNE plot here[0] is just pre-calculated x/y data stored as two attributes. Currently there is an issue with fully integrating the viewer into such a work-flow: for performance reasons, it caches accessed data from the file. This cache needs to be refreshed manually. Sten recently added library support for keeping track of file modifications[1]. That enables me to make the viewer automatically update stale cache whenever a file is modified, making it even easier to integrate. Currently working on that. There's still a ton of polishing and bug-fixing to do. Feedback, suggestions and help are always welcome! [0] http://loom.linnarssonlab.org/dataset/cells/Dentate%20gyrus/dentate_gyrus_C_10X_V2.loom/ http://loom.linnarssonlab.org/dataset/cells/Dentate%20gyrus/... [1] https://github.com/linnarsson-lab/loompy/issues/26 https://github.com/linnarsson-lab/loompy/issues/26
- wowbot 9y ago
  What's the difference between this and jupyter? besides the file format itself obviously
  - 9y ago

by stinos 9y ago

c12 9y ago

I think this is a good trend. The opening of raw data AND the code used to analyse it will aid in others being able to find errors sooner.

vanderZwan 9y ago

This is exactly the motivation behind the Loom file format[0] being developed and used in the molecular neurobiology-group I work for[1]. It is an HDF5-based file format for 'omics data, to deal with the ever-growing size of the data sets. I specifically work on the Loom Viewer[1], a SPA that we're trying to design in such a way that it will be really easy and cheap for research groups to host and share these loom files themselves. This would make easy for other groups to ask simple questions about each others data, and in the worst case the raw loom file is always available for download. We're already hosting some of our own published datasets with this viewer, you can check it out here[3]. To lower expectations a bit: the viewer is not trying to be comparable to the big atlases like [4] or [5] (I mean, it's being developed by one dude - me - so by comparison it's a no-budget OSS project). It's much simpler and basic - the idea is that if you use the Loom file format in your pipeline in a sensible manner, the viewer will more or less know what to do with the data. [0] https://github.com/linnarsson-lab/loompy https://github.com/linnarsson-lab/loompy [1] http://linnarssonlab.org/ http://linnarssonlab.org/ [2] https://github.com/linnarsson-lab/loom-viewer https://github.com/linnarsson-lab/loom-viewer [3] http://loom.linnarssonlab.org/ http://loom.linnarssonlab.org/ [4] http://mouse.brain-map.org/agea http://mouse.brain-map.org/agea [5] https://www.proteinatlas.org/tissue https://www.proteinatlas.org/tissue

reallymental 9y ago
This is a significant amount of effort, and has some very good touches. Especially the Loom-Viewer, I really liked the ease with which one's workflow can integrate into the data analysis.
- vanderZwan 9y ago
  Thanks! :) It's always good to hear that we're on the right track with usability. It's one of the things that I'm most worried about: I've been working pretty isolated for the last year and a half, and lack a background in biology or bioinformatics, and did not even have webdev experience when I took on this project (plenty of embarrassing proof of that in the code). Kudos to Sten Linnarsson, the PI of the group and my boss, for taking a gamble and hiring me anyway. > I really liked the ease with which one's workflow can integrate into the data analysis. Just to make this clear: the file format is a "dumb" data storage, and the viewer a "dumb" plotter of that data. To do more in-depth analysis requires loading the file in python, R, or anything else that might support the files in the future. The idea is to then store the results of this as attributes in the file. For example, the tSNE plot here[0] is just pre-calculated x/y data stored as two attributes. Currently there is an issue with fully integrating the viewer into such a work-flow: for performance reasons, it caches accessed data from the file. This cache needs to be refreshed manually. Sten recently added library support for keeping track of file modifications[1]. That enables me to make the viewer automatically update stale cache whenever a file is modified, making it even easier to integrate. Currently working on that. There's still a ton of polishing and bug-fixing to do. Feedback, suggestions and help are always welcome! [0] http://loom.linnarssonlab.org/dataset/cells/Dentate%20gyrus/dentate_gyrus_C_10X_V2.loom/ http://loom.linnarssonlab.org/dataset/cells/Dentate%20gyrus/... [1] https://github.com/linnarsson-lab/loompy/issues/26 https://github.com/linnarsson-lab/loompy/issues/26
wowbot 9y ago
What's the difference between this and jupyter? besides the file format itself obviously
- 9y ago