5 ms·

biochemist here... I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. Here's t

by toufka 15y ago

biochemist here... I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. If I use a microscope to take images it can be in a Nikon file, a Canon file, a Zeiss file, or a number of other files. None of which are interoperable. TIF, in general, doesn't cut it - it doesn't communicate with the scope and generally can't store all the meta-data that must be kept with the image itself. If I were to dump my raw Nikon files it would help nothing - no one could read them without the $10k software that comes with the scope. Open standards, even for the more commonly used scientific formats, simply don't really exist. And when they do, they're still hacky and ugly. If someone would write some good open-source microscopy, genetic, and basic mathematical software it would make thousands of grad-students' lives easier. Someone PLEASE write a good plasmid viewer (an editor if you're feeling kind) for genbank files. An iPhoto/iTunes for microscopy images. A GUI for basic/common perl/python scripts. Some mathematical software that's not impenetrable (a la MatLab/Mathematica). If these softwares were out there and used, it'd be trivial to dump the raw data and allow anyone to use them. (I'd love to have you guys make me some sweet software. It would take you a relatively short time - these things are standard and already spec'ed out. But sorry - we have no money - we can't really pay you.)

rflrob 15y ago
>I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. > >Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. Release the data and code anyways. I feel your pain with the proprietary formats, but ImageJ is pretty good about dealing with bizarre formats (and other scientists are more likely to have access to the $10k software, or at least know someone who does). I feel your pain with the hacked together scripts. After sharing an analysis program with some collaborators, I had to go through about three rounds of testing to figure out all* of the unreliable assumptions I had made based on one particularly well-formed dataset. This process will make your code better, though, and it will make you a better coder. It will also help other people who are, like you and me, muddling through, trying to find any kind of usable starting point. [*] Okay, "some"
kaarlo_n 15y ago
> Some mathematical software that's not impenetrable (a la MatLab/Mathematica). The Sage project is what you're after: http://www.sagemath.org/ http://www.sagemath.org/
eric_bullington 15y ago
As a practicing scientist interested in open source scientific software, you should consider learning Python if you don't know it already. If you do know some Python, then take a hard look at projects like: [1] Sage: http://www.sagemath.org/ http://www.sagemath.org/ (Sage is scientific software that bundles many of the packages outlined below) [2] SciPy: http://www.scipy.org/ http://www.scipy.org/ Open-source software for mathematics, science, and engineering. [3] NumPy: http://numpy.scipy.org/ http://numpy.scipy.org/ Linear algebra for Python. [4] Scikit-learn: http://scikit-learn.org/stable/ http://scikit-learn.org/stable/ Machine learning in Python. [5] Matplotlib: http://matplotlib.sourceforge.net/ http://matplotlib.sourceforge.net/ Python graphing and plotting library. [6] RPy: http://rpy.sourceforge.net/ http://rpy.sourceforge.net/ Python bindings to the open source R statistical package (modeled after and compatible with S statistical software), which opens up the vast world of statistical computing to Python. As a chemist, you might also be interested in some of the following Python packages, mentioned in a blog post of Python for chemists [1]: Cheminformatics OpenBabel (Pybel), RDKit, OEChem, Daylight (PyDaylight), Cambios Molecular Toolkit, Frowns, PyBabel and MolKit (both part of MGLTools) Computational chemistry OpenBabel, cclib, QMForge, GaussSum, PyQuante, NWChem, Maestro/Jaguar, MMTK Visualisation CCP1GUI, PyMOL, PMV, Zeobuilder, Chimera, VMD, Avogadro Most of the packages listed above feature performance-critical code written in optimized C or Fortran, so they run fast -- much faster than most equivalent proprietary platforms. Really, if you've not looked closely at what Python has to offer, please do yourself a favor and take a close look. If you already know a programming language, you could probably be using Python comfortably in a week or two. 1. Python -- the scripting language of chemistry. http://baoilleach.blogspot.com/2008/03/python-scripting-language-of-chemistry.html http://baoilleach.blogspot.com/2008/03/python-scripting-lang...
yummyfajitas 15y ago
Just curious, some of these seem like really interesting potential OSS projects. In particular, I'd consider writing a couple of them just for practice - i.e., I might write your plasmid viewer/editor just as a project to learn clojure. I imagine I'm not the only one. Why don't you post a more detailed list? The OSS community might surprise you.
- toufka 15y ago
  I guarantee that the best open-source plasmid editor/viewer will become a ubiquitous piece of software in every university. And eventually, even, homes. http://www.mekentosj.com/science/enzymex http://www.mekentosj.com/science/enzymex - guys who made papers made part of it. Nicest interface, but missing lots of features. http://biologylabs.utah.edu/jorgensen/wayned/ape/ http://biologylabs.utah.edu/jorgensen/wayned/ape/ - is what everyone uses. It's old. Functional, but missing SO much of what's possible. https://www.dna20.com/genedesigner2/ https://www.dna20.com/genedesigner2/ - made by the best in the industry. Look at that shit. http://bioclipse.net/screenshots http://bioclipse.net/screenshots - opensource, but look how difficult. How not right. The rules are straightforward. It's a lot like writing out an API. A good manipulator of well-documented data structures.
  - yummyfajitas 15y ago
    Please write to me, I'd love to discuss this with you in more detail. Contact info is in my profile.
mironathetin 15y ago
"these things are standard and already spec'ed out" Oh my, forget it. I am a scientist (astronomy) and I write software for my project. Its all standard, but EVERY single astronomer that you ask, has a different interpretation of the standards. And everybody has his/her own special reason, why he/she prefers some hacked code from the internet to our beautifully perfect software that even provides guis for standard work. Either it is crappy, or it requires too many resources. And then, scientists don't talk. Our colleagues are sitting in their office, waiting for software developers who can read their minds (it's all standard). It is incredibly hard to get good requirements that are accepted by more than one scientist as good requirements.
RyanMcGreal 15y ago
> Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. All excellent reasons to move to open source!

by toufka 15y ago

rflrob 15y ago

>I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. > >Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. Release the data and code anyways. I feel your pain with the proprietary formats, but ImageJ is pretty good about dealing with bizarre formats (and other scientists are more likely to have access to the $10k software, or at least know someone who does). I feel your pain with the hacked together scripts. After sharing an analysis program with some collaborators, I had to go through about three rounds of testing to figure out all* of the unreliable assumptions I had made based on one particularly well-formed dataset. This process will make your code better, though, and it will make you a better coder. It will also help other people who are, like you and me, muddling through, trying to find any kind of usable starting point. [*] Okay, "some"

kaarlo_n 15y ago

> Some mathematical software that's not impenetrable (a la MatLab/Mathematica). The Sage project is what you're after: http://www.sagemath.org/ http://www.sagemath.org/

eric_bullington 15y ago

As a practicing scientist interested in open source scientific software, you should consider learning Python if you don't know it already. If you do know some Python, then take a hard look at projects like: [1] Sage: http://www.sagemath.org/ http://www.sagemath.org/ (Sage is scientific software that bundles many of the packages outlined below) [2] SciPy: http://www.scipy.org/ http://www.scipy.org/ Open-source software for mathematics, science, and engineering. [3] NumPy: http://numpy.scipy.org/ http://numpy.scipy.org/ Linear algebra for Python. [4] Scikit-learn: http://scikit-learn.org/stable/ http://scikit-learn.org/stable/ Machine learning in Python. [5] Matplotlib: http://matplotlib.sourceforge.net/ http://matplotlib.sourceforge.net/ Python graphing and plotting library. [6] RPy: http://rpy.sourceforge.net/ http://rpy.sourceforge.net/ Python bindings to the open source R statistical package (modeled after and compatible with S statistical software), which opens up the vast world of statistical computing to Python. As a chemist, you might also be interested in some of the following Python packages, mentioned in a blog post of Python for chemists [1]: Cheminformatics OpenBabel (Pybel), RDKit, OEChem, Daylight (PyDaylight), Cambios Molecular Toolkit, Frowns, PyBabel and MolKit (both part of MGLTools) Computational chemistry OpenBabel, cclib, QMForge, GaussSum, PyQuante, NWChem, Maestro/Jaguar, MMTK Visualisation CCP1GUI, PyMOL, PMV, Zeobuilder, Chimera, VMD, Avogadro Most of the packages listed above feature performance-critical code written in optimized C or Fortran, so they run fast -- much faster than most equivalent proprietary platforms. Really, if you've not looked closely at what Python has to offer, please do yourself a favor and take a close look. If you already know a programming language, you could probably be using Python comfortably in a week or two. 1. Python -- the scripting language of chemistry. http://baoilleach.blogspot.com/2008/03/python-scripting-language-of-chemistry.html http://baoilleach.blogspot.com/2008/03/python-scripting-lang...

yummyfajitas 15y ago

Just curious, some of these seem like really interesting potential OSS projects. In particular, I'd consider writing a couple of them just for practice - i.e., I might write your plasmid viewer/editor just as a project to learn clojure. I imagine I'm not the only one. Why don't you post a more detailed list? The OSS community might surprise you.

toufka 15y ago
I guarantee that the best open-source plasmid editor/viewer will become a ubiquitous piece of software in every university. And eventually, even, homes. http://www.mekentosj.com/science/enzymex http://www.mekentosj.com/science/enzymex - guys who made papers made part of it. Nicest interface, but missing lots of features. http://biologylabs.utah.edu/jorgensen/wayned/ape/ http://biologylabs.utah.edu/jorgensen/wayned/ape/ - is what everyone uses. It's old. Functional, but missing SO much of what's possible. https://www.dna20.com/genedesigner2/ https://www.dna20.com/genedesigner2/ - made by the best in the industry. Look at that shit. http://bioclipse.net/screenshots http://bioclipse.net/screenshots - opensource, but look how difficult. How not right. The rules are straightforward. It's a lot like writing out an API. A good manipulator of well-documented data structures.
- yummyfajitas 15y ago
  Please write to me, I'd love to discuss this with you in more detail. Contact info is in my profile.

mironathetin 15y ago

"these things are standard and already spec'ed out" Oh my, forget it. I am a scientist (astronomy) and I write software for my project. Its all standard, but EVERY single astronomer that you ask, has a different interpretation of the standards. And everybody has his/her own special reason, why he/she prefers some hacked code from the internet to our beautifully perfect software that even provides guis for standard work. Either it is crappy, or it requires too many resources. And then, scientists don't talk. Our colleagues are sitting in their office, waiting for software developers who can read their minds (it's all standard). It is incredibly hard to get good requirements that are accepted by more than one scientist as good requirements.

RyanMcGreal 15y ago

> Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. All excellent reasons to move to open source!