5 ms·

Short answer: Yes, my 30 year old Fortran code runs (with a few minor edits between f77 and modern fortran), as did my ancient Perl codes. Watching the density

by hpcjoe 6y ago

Short answer: Yes, my 30 year old Fortran code runs (with a few minor edits between f77 and modern fortran), as did my ancient Perl codes. Watching the density functional theory based molecular dynamics zip along at ~2 seconds per time step on my 2 year old laptop, versus the roughly 6k seconds per time step on an old Sun machine back in 1991. I remember the same code getting down to 60 seconds per time step on my desktop R8k machine in the late 90s. Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. Perl code that worked with big XML file input in the mid 2000s continues to work, though I've largely abandoned using XML for data interchange. C code I wrote in the mid 90s compiled, albeit with errors that needed to be corrected. C++ code was less forgiving. Over the past 4 months, I had to forward port a code from Boost 1.41 to Boost 1.65. Enough changes over 9 years (code was from 2011) that it presented a problem. So I had to follow the changes in the API and fix it. I am quite thankful I've avoided the various fads in platforms and languages over the years. Keep inputs in simple textual format that can be trivially parsed.

atrettel 6y ago
> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianness changes you can still read it decades later. I've been writing a computational fluid dynamics code recently and decided to only write binary output for those reasons. I'm not convinced of the long-term stability of other formats. I've seen colleagues struggle to read data in proprietary formats even a few years after creating it. Binary is just simple and avoids all of that. Anybody can read it if needed.
- petschge 6y ago
  Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readability of the data for half a century).
  - atrettel 6y ago
    Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over the proprietary format that they did end up using. But HDF5 requires an additional library. I did not want to use any external libraries in my particular project (other than MPI), so I tried to look for a solution that achieves close to what HDF5 can achieve but without requiring something as "heavy" as HDF5. I have to admit that perhaps my design choice does not work for more complex situations, but I think it is something people should consider before tying themselves down too much. [1] http://www.xdmf.org/index.php/Main_Page http://www.xdmf.org/index.php/Main_Page
    - petschge 6y ago
      Having an Xdmf file alongside is nice, but the breaking changes between v2 and v3 are very annoying. And I understand the want to have few external dependencies, but at least HDF5 is straight forward to compile and available as a pre-compiled module on all supercomputers that I have ever seen.
  - hpcjoe 6y ago
    I got into the habit of documenting each file with a file.meta that I could view later on. I did binary dumps in the past because ascii dumps (remember, 90s) were far more time/space expensive. HDF wasn't quite an option then, either HDF4, or HDF5. These days I would probably look at something like that, though, to be honest, there is always a danger of choosing something that may not be supported over the long term. This is why I generally prefer open and simple formats for everything. HDF5 is nice and open. One needs to look carefully at the total risk of using a proprietary format/system for any part of their storage. Chances are you will not be able to even read older data within a small number of decades if any of the format/system dependent technologies goes away. I've got old word processor files from the mid 80s, that I can't read. What I've written there (mostly college papers) is lost (which may be a net positive for humanity). My tarballs, and zip files though, are readable 30+ years later. That is pretty amazing. Simple, documented, and open formats. Picture a time when you can't read/open your pptx/xlsx/docx files any more. Same with data. Simple binary formats are like CSV files, but you do need to maintain metadata on their contents, and document it extensively in the code as to what you are reading/writing, why you are doing this, and how you are doing this. I think this will get more important over time as we start asking questions on how to maintain open artefact repositories for data and code. The fewer dependencies the better. And unlike the recent gene renaming snafu in biology[1], you really, never, want your tool to get in the way of the science. Either in terms of formats, or interpretation of data. [1] https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Rochus 6y ago
Yes, I know a couple of Fortran 77 apps and libraries which were developed more than 25 years ago and which are still in use today. My C++ Qt GUI application for NMR spectrum analysis (https://github.com/rochus-keller/CARA https://github.com/rochus-keller/CARA) runs since 20 years now with continuing high download and citation rates. So obviously C++/Qt or Fortran 77 are very well suited to outlast time.
- O_H_E 6y ago
  Nice. Interesting to know that Github starts aren't always a representative metric.
  - Rochus 6y ago
    Yes, many of my apps and libs were more than ten years old when I pushed them to github. Some projects started before git was invented.

by hpcjoe 6y ago

atrettel 6y ago

> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianness changes you can still read it decades later. I've been writing a computational fluid dynamics code recently and decided to only write binary output for those reasons. I'm not convinced of the long-term stability of other formats. I've seen colleagues struggle to read data in proprietary formats even a few years after creating it. Binary is just simple and avoids all of that. Anybody can read it if needed.

petschge 6y ago
Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readability of the data for half a century).
- atrettel 6y ago
  Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over the proprietary format that they did end up using. But HDF5 requires an additional library. I did not want to use any external libraries in my particular project (other than MPI), so I tried to look for a solution that achieves close to what HDF5 can achieve but without requiring something as "heavy" as HDF5. I have to admit that perhaps my design choice does not work for more complex situations, but I think it is something people should consider before tying themselves down too much. [1] http://www.xdmf.org/index.php/Main_Page http://www.xdmf.org/index.php/Main_Page
  - petschge 6y ago
    Having an Xdmf file alongside is nice, but the breaking changes between v2 and v3 are very annoying. And I understand the want to have few external dependencies, but at least HDF5 is straight forward to compile and available as a pre-compiled module on all supercomputers that I have ever seen.
- hpcjoe 6y ago
  I got into the habit of documenting each file with a file.meta that I could view later on. I did binary dumps in the past because ascii dumps (remember, 90s) were far more time/space expensive. HDF wasn't quite an option then, either HDF4, or HDF5. These days I would probably look at something like that, though, to be honest, there is always a danger of choosing something that may not be supported over the long term. This is why I generally prefer open and simple formats for everything. HDF5 is nice and open. One needs to look carefully at the total risk of using a proprietary format/system for any part of their storage. Chances are you will not be able to even read older data within a small number of decades if any of the format/system dependent technologies goes away. I've got old word processor files from the mid 80s, that I can't read. What I've written there (mostly college papers) is lost (which may be a net positive for humanity). My tarballs, and zip files though, are readable 30+ years later. That is pretty amazing. Simple, documented, and open formats. Picture a time when you can't read/open your pptx/xlsx/docx files any more. Same with data. Simple binary formats are like CSV files, but you do need to maintain metadata on their contents, and document it extensively in the code as to what you are reading/writing, why you are doing this, and how you are doing this. I think this will get more important over time as we start asking questions on how to maintain open artefact repositories for data and code. The fewer dependencies the better. And unlike the recent gene renaming snafu in biology[1], you really, never, want your tool to get in the way of the science. Either in terms of formats, or interpretation of data. [1] https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Rochus 6y ago

Yes, I know a couple of Fortran 77 apps and libraries which were developed more than 25 years ago and which are still in use today. My C++ Qt GUI application for NMR spectrum analysis (https://github.com/rochus-keller/CARA https://github.com/rochus-keller/CARA) runs since 20 years now with continuing high download and citation rates. So obviously C++/Qt or Fortran 77 are very well suited to outlast time.

O_H_E 6y ago
Nice. Interesting to know that Github starts aren't always a representative metric.
- Rochus 6y ago
  Yes, many of my apps and libs were more than ten years old when I pushed them to github. Some projects started before git was invented.