5 ms·

Shameless plug: In January I'll be focusing on a project which deals specifically with scientific software: http://sciencetoolbox.org/ http://sciencetoolbox.org

Shameless plug: In January I'll be focusing on a project which deals specifically with scientific software: http://sciencetoolbox.org/ http://sciencetoolbox.org/ This current version is a product of a hackathon, but this month will be improving it and adding functionality which brings the scientific software developer and her efforts into focus. Scientific software is gaining importance, but the recognition its developers get is trailing behind - I want to raise the level of associated recognition/prestige (among other related things). Some other projects rely on data collected here, e.g. a recommendation engine for said software that enhances GitHub: http://juretriglav.si/discovery-of-scientific-software/ http://juretriglav.si/discovery-of-scientific-software/ Shameless plug continues: if you'd like to keep track of what I'm doing I suggest you either follow the project on GitHub (https://github.com/ScienceToolbox/sciencetoolbox https://github.com/ScienceToolbox/sciencetoolbox) or Twitter (https://twitter.com/sciencetoolbox https://twitter.com/sciencetoolbox).

toufka 12y ago
Units. I'd love so much for a standard fileformat/interpreter/concept that contained SI units as a requirement for most datatypes. Much to be learned from my TI-89.
- dalke 12y ago
  I don't know what you mean by "as a requirement for most datatypes". I work with molecules, where distances are typically measured in Angstroms and masses in amu. I don't want to have factors of 1E−10 and 1.660538922E−27 hanging around my code.
  - detaro 12y ago
    I assume that he means that the type system knows that these are in useful units, and tracks them, so you can have it check that your calculation actually results in a value of the unit you expected.
    - wtallis 12y ago
      Yeah. There's really no reason why our hot new scientific computing languages and libraries should all be lacking capabilities that graphing calculators had in the 1980s when they only had 2k of RAM. HP demonstrated that it doesn't even require a CAS to be extremely useful.
      - dalke 12y ago
        I know what unit libraries are, and why they can be useful. There are several units libraries for Python, the Boost C++ library includes one, etc. I don't know why they should be a 'requirement for most datatypes'. Could someone please explain the requirement part? As a further clarification, why should people in scientific computing, in fields which use non-SI units like eV, amu, Angstrom, barn, light year, and megaparsec, use a programming language which requires SI units? Quasar 3C 273 is 749 megaparsecs from us, or 2.31E25 meters away. I don't see why SI should be preferred.
        wtallis 12y ago
        Any programming language that makes units a first class part of its type system would allow for defining custom units just as easily as defining other data types. Nobody's saying that SI has to be the only units expressible, just that it has to be the foundation of the unit system. Likewise, unitless numerical quantities will necessarily still be expressible, but using those types for variables that represent a value with a unit should be considered extremely poor practice, just like when communicating those values on paper. There's really not a good reason to argue against using only SI units for data interchange formats. It's trivial to map to the preferred units on import or display, if and only if you know what units the input data is in. I've dealt with too many bugs where interacting programs have differing assumptions about meters, centimeters, and millimeters to believe that the flexibility of storing different units on disk is ever worth the trouble.
        dalke 12y ago
        I use Python. If I use one of the third-party packages for units then I can do what you say I can, but at extreme cost. Every single operation checks for unit conversion, on the off-chance that the values aren't compatible. The system, in trying to be nice to me, ends up making things invisibly slow. (In practice, the performance code runs in C, so the Python/C boundary would have to negotiate the array types for full unit safety.) In my work the base unit of length is angstroms. I've used nanometers a few times, and never used any other length unit, though I know that GROMAC's xtc format uses picometers. Saying something has a volume of 600 cubic angstroms is much more useful than 6E-28 cubic meters. While I can appreciate that other fields closer to human scale use may like to standardize through SI, I don't want your preferences enforced on my field. All I see is the chance to make things worse, and slower, and don't see any advantages. One of my data formats has coordinates in angstroms, like "8.420 50.899 85.486". How would you suggest that I write that in an exchange format? As "8.420E-10 50.899E-10 85.486E-10"? (Or the last two normalized to E-11.) At the very least that's a lot of data for very little gain. It gets worse for trajectories, which might save 1 million time steps x 10,000 atoms/time step x 3 coordinates/atom = 3 billion coordinates to an exchange file. I see no advantage to doing that in SI units. In practice those distance coordinates will likely internally represented in angstroms. Consider that the Lennard-Jones potential is sometimes written as A/r^12 - B/r^6 , with expected values of r around 1E-10m. The denominator of the first will go to 1E-120 in intermediate form, and not be representable in 32-bit float. While not relevant for Python, which uses 64 bit floats, some molecular dynamics programs will use 32 bit float. (Eg, for older GPU machines, or to save space.) My other example was the atomic mass unit, another non-SI unit. I have only used amu (for chemistry) or dalton (for biology) in my work, not kilograms. It seems pointless to require that I store the mass of a carbon as 1.9926467051999998e-26 kg instead of 12 amu. I therefore disagree, and believe there are good reasons to argue against SI units for some data interchange formats. I agree that I want to store a single distance unit on disk, only that unit is the non-SI unit angstrom and amu, and not the tremendously huge meter or kg.
  - toufka 12y ago
    The idea that you can export a csv file or excel worksheet full of unitless numbers is scientifically whacky. In the same way most would scoff at an graph without axis labels why too don't we scoff at a data file without units.
    - dalke 12y ago
      What's whacky about implicit units for a given format? I use data files without explicit units all the time. A format specification, which might be explicit or implicit, might say that a given file is stored in CSV format where the first column contains a molecule representation (in my case as a SMILES string), followed by an identifier, followed by the molecular weight (in amu, which is the only reasonable unit), followed by surface area (in Å², which is always the case), followed by volume (in Å³, again, always the case). You'll note that none of these are SI units. I have another file containing a molecular structure in SDF format, which starts: 16125001 -OEChem-04231101242D 44 45 0 1 0 0 0 0 0999 V2000 4.5411 4.0194 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0 3.6750 2.5194 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0 Line 4, the 44 is "number of atoms", the 45 is "number of bonds", and the (4.5411, 4.0194, 0.0000) is a coordinate in angstroms. I see a distinct lack of units in the data file. I have another file, in PDB format, containing lines like: ATOM 34 N GLY 1 6 8.420 50.899 85.486 0.50 51.30 The (8.420, 50.899, 85.486) is a coordinate in Å. The other numbers are identifiers of one sort or other, or the unitless occupancy and B-factor. Are you really going to scoff at my entire field, for working with data files without explicit units since the late 1960s? When working specifically with Excel files, an organization tends to stabilize on what certain column titles mean, so the a title of "MW" means "molecular weight in AMU", etc. This is more complicated when applied to values which are parameter dependent (charge at pH 7.5 vs. 6.0), or depend on specific models and/or software version. You'll notice that pH is a unitless number. Units are only a subset of the ontology usually omitted from a given file format. Others include unitless terms like B-factor, pH, "number of atoms", prediction model, and implementation version. This ontology is often instead made explicit in external format documentation or through shared knowledge of the users of that data file. What advantage there is for me or my field to have 'SI units as a requirement for most datatypes', especially since we often deal with non-SI units like Å and amu? I don't see anything except more confusion, more chances for error, and performance overhead of going from/to SI units instead of staying in the domain-specific preferred system. The people who I've seen try to use a Semantic Web/Linked Data approach end up bogged down in verbose and slow to parse data formats that make it hard to do real work, because the software has to be wary that the input one moment might be in Å, the next in nm, the third in m, and the fourth in yoctoparsecs.
- nitrogen 12y ago
  I loved the unit conversions on my '89. You might enjoy checking out the units command line tool, or the frink programming language. http://linux.die.net/man/1/units http://linux.die.net/man/1/units http://futureboy.us/frinkdocs/ http://futureboy.us/frinkdocs/
stared 12y ago
How about sorting by stars? And how to add things - I am trying, but without a success?
- juretriglav 12y ago
  Like I said, the current minimal product is hacked together from various pieces (scrapers, graph database generators, API wrappers, etc.) and is far from what it will be in a few weeks. Things like adding were changed from manual to automatic (based on citations), and some of the interface remained the same. I suggest you check back with http://sciencetoolbox.org http://sciencetoolbox.org at the end of January. Feature list: Full-text search through the software (also by metadata, tags, citations) More extensive/up-to-date indexing of citations User profiles (for owners/developers and users) Citation notifications Usage visualizations (graph network of cited software, percent citations in a given category and much more) Should be fun :)
  - stared 12y ago
    Good luck with this project! As a (ex?)scientist, the problem I remember is looking if there is a mature-enough library and still in active development/support. (Usually there are many, but only a few actually useful.) A few other ways to go is to scan arXiv for citations of software, https://pypi.python.org/pypi https://pypi.python.org/pypi for scientific packages, etc.

toufka 12y ago

Units. I'd love so much for a standard fileformat/interpreter/concept that contained SI units as a requirement for most datatypes. Much to be learned from my TI-89.

dalke 12y ago
I don't know what you mean by "as a requirement for most datatypes". I work with molecules, where distances are typically measured in Angstroms and masses in amu. I don't want to have factors of 1E−10 and 1.660538922E−27 hanging around my code.
- detaro 12y ago
  I assume that he means that the type system knows that these are in useful units, and tracks them, so you can have it check that your calculation actually results in a value of the unit you expected.
  - wtallis 12y ago
    Yeah. There's really no reason why our hot new scientific computing languages and libraries should all be lacking capabilities that graphing calculators had in the 1980s when they only had 2k of RAM. HP demonstrated that it doesn't even require a CAS to be extremely useful.
    - dalke 12y ago
      I know what unit libraries are, and why they can be useful. There are several units libraries for Python, the Boost C++ library includes one, etc. I don't know why they should be a 'requirement for most datatypes'. Could someone please explain the requirement part? As a further clarification, why should people in scientific computing, in fields which use non-SI units like eV, amu, Angstrom, barn, light year, and megaparsec, use a programming language which requires SI units? Quasar 3C 273 is 749 megaparsecs from us, or 2.31E25 meters away. I don't see why SI should be preferred.
      - wtallis 12y ago
        Any programming language that makes units a first class part of its type system would allow for defining custom units just as easily as defining other data types. Nobody's saying that SI has to be the only units expressible, just that it has to be the foundation of the unit system. Likewise, unitless numerical quantities will necessarily still be expressible, but using those types for variables that represent a value with a unit should be considered extremely poor practice, just like when communicating those values on paper. There's really not a good reason to argue against using only SI units for data interchange formats. It's trivial to map to the preferred units on import or display, if and only if you know what units the input data is in. I've dealt with too many bugs where interacting programs have differing assumptions about meters, centimeters, and millimeters to believe that the flexibility of storing different units on disk is ever worth the trouble.
        dalke 12y ago
        I use Python. If I use one of the third-party packages for units then I can do what you say I can, but at extreme cost. Every single operation checks for unit conversion, on the off-chance that the values aren't compatible. The system, in trying to be nice to me, ends up making things invisibly slow. (In practice, the performance code runs in C, so the Python/C boundary would have to negotiate the array types for full unit safety.) In my work the base unit of length is angstroms. I've used nanometers a few times, and never used any other length unit, though I know that GROMAC's xtc format uses picometers. Saying something has a volume of 600 cubic angstroms is much more useful than 6E-28 cubic meters. While I can appreciate that other fields closer to human scale use may like to standardize through SI, I don't want your preferences enforced on my field. All I see is the chance to make things worse, and slower, and don't see any advantages. One of my data formats has coordinates in angstroms, like "8.420 50.899 85.486". How would you suggest that I write that in an exchange format? As "8.420E-10 50.899E-10 85.486E-10"? (Or the last two normalized to E-11.) At the very least that's a lot of data for very little gain. It gets worse for trajectories, which might save 1 million time steps x 10,000 atoms/time step x 3 coordinates/atom = 3 billion coordinates to an exchange file. I see no advantage to doing that in SI units. In practice those distance coordinates will likely internally represented in angstroms. Consider that the Lennard-Jones potential is sometimes written as A/r^12 - B/r^6 , with expected values of r around 1E-10m. The denominator of the first will go to 1E-120 in intermediate form, and not be representable in 32-bit float. While not relevant for Python, which uses 64 bit floats, some molecular dynamics programs will use 32 bit float. (Eg, for older GPU machines, or to save space.) My other example was the atomic mass unit, another non-SI unit. I have only used amu (for chemistry) or dalton (for biology) in my work, not kilograms. It seems pointless to require that I store the mass of a carbon as 1.9926467051999998e-26 kg instead of 12 amu. I therefore disagree, and believe there are good reasons to argue against SI units for some data interchange formats. I agree that I want to store a single distance unit on disk, only that unit is the non-SI unit angstrom and amu, and not the tremendously huge meter or kg.
- toufka 12y ago
  The idea that you can export a csv file or excel worksheet full of unitless numbers is scientifically whacky. In the same way most would scoff at an graph without axis labels why too don't we scoff at a data file without units.
  - dalke 12y ago
    What's whacky about implicit units for a given format? I use data files without explicit units all the time. A format specification, which might be explicit or implicit, might say that a given file is stored in CSV format where the first column contains a molecule representation (in my case as a SMILES string), followed by an identifier, followed by the molecular weight (in amu, which is the only reasonable unit), followed by surface area (in Å², which is always the case), followed by volume (in Å³, again, always the case). You'll note that none of these are SI units. I have another file containing a molecular structure in SDF format, which starts: 16125001 -OEChem-04231101242D 44 45 0 1 0 0 0 0 0999 V2000 4.5411 4.0194 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0 3.6750 2.5194 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0 Line 4, the 44 is "number of atoms", the 45 is "number of bonds", and the (4.5411, 4.0194, 0.0000) is a coordinate in angstroms. I see a distinct lack of units in the data file. I have another file, in PDB format, containing lines like: ATOM 34 N GLY 1 6 8.420 50.899 85.486 0.50 51.30 The (8.420, 50.899, 85.486) is a coordinate in Å. The other numbers are identifiers of one sort or other, or the unitless occupancy and B-factor. Are you really going to scoff at my entire field, for working with data files without explicit units since the late 1960s? When working specifically with Excel files, an organization tends to stabilize on what certain column titles mean, so the a title of "MW" means "molecular weight in AMU", etc. This is more complicated when applied to values which are parameter dependent (charge at pH 7.5 vs. 6.0), or depend on specific models and/or software version. You'll notice that pH is a unitless number. Units are only a subset of the ontology usually omitted from a given file format. Others include unitless terms like B-factor, pH, "number of atoms", prediction model, and implementation version. This ontology is often instead made explicit in external format documentation or through shared knowledge of the users of that data file. What advantage there is for me or my field to have 'SI units as a requirement for most datatypes', especially since we often deal with non-SI units like Å and amu? I don't see anything except more confusion, more chances for error, and performance overhead of going from/to SI units instead of staying in the domain-specific preferred system. The people who I've seen try to use a Semantic Web/Linked Data approach end up bogged down in verbose and slow to parse data formats that make it hard to do real work, because the software has to be wary that the input one moment might be in Å, the next in nm, the third in m, and the fourth in yoctoparsecs.
nitrogen 12y ago
I loved the unit conversions on my '89. You might enjoy checking out the units command line tool, or the frink programming language. http://linux.die.net/man/1/units http://linux.die.net/man/1/units http://futureboy.us/frinkdocs/ http://futureboy.us/frinkdocs/

stared 12y ago

How about sorting by stars? And how to add things - I am trying, but without a success?

juretriglav 12y ago
Like I said, the current minimal product is hacked together from various pieces (scrapers, graph database generators, API wrappers, etc.) and is far from what it will be in a few weeks. Things like adding were changed from manual to automatic (based on citations), and some of the interface remained the same. I suggest you check back with http://sciencetoolbox.org http://sciencetoolbox.org at the end of January. Feature list: Full-text search through the software (also by metadata, tags, citations) More extensive/up-to-date indexing of citations User profiles (for owners/developers and users) Citation notifications Usage visualizations (graph network of cited software, percent citations in a given category and much more) Should be fun :)
- stared 12y ago
  Good luck with this project! As a (ex?)scientist, the problem I remember is looking if there is a mature-enough library and still in active development/support. (Usually there are many, but only a few actually useful.) A few other ways to go is to scan arXiv for citations of software, https://pypi.python.org/pypi https://pypi.python.org/pypi for scientific packages, etc.