3 ms·

I think the author is running afoul of a very common problem (albeit he seems to be slightly more aware of it than most): "Ease of Use" and "Ease of Learning" a

I think the author is running afoul of a very common problem (albeit he seems to be slightly more aware of it than most): "Ease of Use" and "Ease of Learning" are two different, and often incompatible goals. He praises the command line utilities as offering productivity improvements of an order of magnitude or more, but then castigates is for not having changed in half a century. His description of the 2 cultures of computing is also interesting because he clearly understands that computer programming is literally the use of language to instruct the computer. He feels the complexity is incidental because it is not directly related to the problem that he wants to solve, but he completely ignores the necessity of learning the language. The command line is a general purpose instrument with an intentionally broad interface. I think it is disingenuous to assume that one could build purpose built tools that do what you want, before you even know what it is you want. The complexity is specifically to afford the flexibility to do whatever you might want to do in the future. It hasn't changed (much) for 50 years because nobody has found a better way to do it. Feel free to improve the state of the art ;-)

a_bonobo 11y ago
> nobody has found a better way to do it. People have found a better way to do it! It's just that research software ignores the better way because it's mostly "proof of concept", once the paper is out, who cares who uses it, that's not my department. The incentive is to publish, not to create usable software. In fact, spending the time to make your software usable is wasted - no funding committee, no tenure committee, no supervisor cares. Here are some of my "command line bullshittery" problems I run into in bioinformatics: - There's no Makefile, just .cc files, you have to figure out how it was compiled. 90% of the times a simple "gcc" won't cut it. - There is a Makefile but the flags are all outdated and there's no ./configure to get flags for your system - There is a Makefile, but it depends on old versions and doesn't say so. The new versions don't work. (samtools v1.0 changed the API and there's a ton of software (BioDB::Sam) out there that assumes a pre-v1.0 API, so you have to override the system-wide installation, great "fun" if you're on a cluster where each node has its own environment and you have no root) - the software depends on other research software that doesn't exist anymore (google-fu to find a forgotten dusty tar.gz somewhere) - It's a binary, but it gives you empty output if you have a typo in your flags (Just last week with Blast2Go: if you use "annot" instead of "-annot" it won't complain, you just get empty output) - it depends on environment variables but it doesn't tell you so, it just crashes when it tries to run a dependency program using $PROGRAM_ROOT/bin/program (even better: MAKER doesn't crash, you just have one error line in about a million lines of STDOUT, chances are you don't realize that it had problems in the first place) - program depends on loading data from disk using MySQL but MySQL wasn't compiled with local-infile=1 I'm sure I've wasted about half of my PhD's time finding out about highly program-specific problems that don't appear anywhere else. It has made me a somewhat proficient debugger but that is lifetime that I could have spent on analyzing actual results & writing papers, things that are actually helpful in my career.
- stragulus 11y ago
  All of those things sound like they are mostly the product of someone before you being lazy and very poor at writing maintainable software or documentation. This argument is akin to blaming carpentry when someone has built a poorly constructed house. Sure, it would be nice if you could spend the majority of your time on the fun stuff, and it looks like your predecessors certainly tried. But in that case you should probably not actually write software and choose a more theoretical career path if you can't be bothered to learn how to use a hammer.
  - a_bonobo 11y ago
    >Sure, it would be nice if you could spend the majority of your time on the fun stuff, and it looks like your predecessors certainly tried. But in that case you should probably not actually write software and choose a more theoretical career path if you can't be bothered to learn how to use a hammer. You didn't understand what OP's post was getting at, you didn't what my post was getting at, yet you still managed to be condescending. Congratulations.
- ddingus 11y ago
  There is an argument here: At one point, where each of those problems lie, somebody prioritized actual results and writing papers over a modest investment in the tools often needed for writing papers and obtaining actual results. Perhaps those modest investments are worth far more than people realize? I'm not sure we can design that away either. There is always the body of tools people depend on and rewriting them is an even more difficult investment for people focused on other things to make! New / better systems and tools could make a lot of sense, but given the use case here, who exactly is going to fund those?
- mikekchar 11y ago
  Hmm... We don't seem to be connecting here. When I said "there is no better way to do it", I meant that interacting with tools through the command line (or shell scripts) is the best way we have of doing these kinds of tasks. I didn't say so explicitly, but my intention was to include all of the usual tools that one would use (grep, awk, make, whatever). Though it was several decades ago, I also suffered greatly from having to deal with poorly built academic projects. It's not even confined to academia. A friend of mine often tells a tale of a start up who spent 3 years building software only to discover that they had built such a mess that they couldn't deploy (never having done it in 3 years... :-P). The company ran out of money before they solved the issue. I may be wrong, but I don't think that's what the original article was complaining about. I think he was frustrated that while the tools were very good, they were hard to teach to his students. He wishes there were easier to learn systems for doing this work. Personally, I don't think it will ever happen because the problem space is quite complicated. You can make it easier to learn by restricting what it can do, but eventually you butt up against that ceiling, making it extremely difficult to use. That's why we are all still using these same tools decades years later (although, I'm quite happy to ditch make in favour of more modern equivalents [1]) I can't tell you the number of times I've wandered into a new project and spent every spare second writing README files, fixing the build systems, removing dead code, etc. I've even worked on projects where there wasn't a single person who knew how to build the entire project -- all they could do was compile individual files and link it to pre-built libraries. They thought me crazy for even wanting to know how to build it. We do our industry a grave disservice by not teaching this stuff somewhere in our education system. [1] Says the guy who just wrote a new make file a couple of weeks ago <whistling>

a_bonobo 11y ago

> nobody has found a better way to do it. People have found a better way to do it! It's just that research software ignores the better way because it's mostly "proof of concept", once the paper is out, who cares who uses it, that's not my department. The incentive is to publish, not to create usable software. In fact, spending the time to make your software usable is wasted - no funding committee, no tenure committee, no supervisor cares. Here are some of my "command line bullshittery" problems I run into in bioinformatics: - There's no Makefile, just .cc files, you have to figure out how it was compiled. 90% of the times a simple "gcc" won't cut it. - There is a Makefile but the flags are all outdated and there's no ./configure to get flags for your system - There is a Makefile, but it depends on old versions and doesn't say so. The new versions don't work. (samtools v1.0 changed the API and there's a ton of software (BioDB::Sam) out there that assumes a pre-v1.0 API, so you have to override the system-wide installation, great "fun" if you're on a cluster where each node has its own environment and you have no root) - the software depends on other research software that doesn't exist anymore (google-fu to find a forgotten dusty tar.gz somewhere) - It's a binary, but it gives you empty output if you have a typo in your flags (Just last week with Blast2Go: if you use "annot" instead of "-annot" it won't complain, you just get empty output) - it depends on environment variables but it doesn't tell you so, it just crashes when it tries to run a dependency program using $PROGRAM_ROOT/bin/program (even better: MAKER doesn't crash, you just have one error line in about a million lines of STDOUT, chances are you don't realize that it had problems in the first place) - program depends on loading data from disk using MySQL but MySQL wasn't compiled with local-infile=1 I'm sure I've wasted about half of my PhD's time finding out about highly program-specific problems that don't appear anywhere else. It has made me a somewhat proficient debugger but that is lifetime that I could have spent on analyzing actual results & writing papers, things that are actually helpful in my career.

stragulus 11y ago
All of those things sound like they are mostly the product of someone before you being lazy and very poor at writing maintainable software or documentation. This argument is akin to blaming carpentry when someone has built a poorly constructed house. Sure, it would be nice if you could spend the majority of your time on the fun stuff, and it looks like your predecessors certainly tried. But in that case you should probably not actually write software and choose a more theoretical career path if you can't be bothered to learn how to use a hammer.
- a_bonobo 11y ago
  >Sure, it would be nice if you could spend the majority of your time on the fun stuff, and it looks like your predecessors certainly tried. But in that case you should probably not actually write software and choose a more theoretical career path if you can't be bothered to learn how to use a hammer. You didn't understand what OP's post was getting at, you didn't what my post was getting at, yet you still managed to be condescending. Congratulations.
ddingus 11y ago
There is an argument here: At one point, where each of those problems lie, somebody prioritized actual results and writing papers over a modest investment in the tools often needed for writing papers and obtaining actual results. Perhaps those modest investments are worth far more than people realize? I'm not sure we can design that away either. There is always the body of tools people depend on and rewriting them is an even more difficult investment for people focused on other things to make! New / better systems and tools could make a lot of sense, but given the use case here, who exactly is going to fund those?
mikekchar 11y ago
Hmm... We don't seem to be connecting here. When I said "there is no better way to do it", I meant that interacting with tools through the command line (or shell scripts) is the best way we have of doing these kinds of tasks. I didn't say so explicitly, but my intention was to include all of the usual tools that one would use (grep, awk, make, whatever). Though it was several decades ago, I also suffered greatly from having to deal with poorly built academic projects. It's not even confined to academia. A friend of mine often tells a tale of a start up who spent 3 years building software only to discover that they had built such a mess that they couldn't deploy (never having done it in 3 years... :-P). The company ran out of money before they solved the issue. I may be wrong, but I don't think that's what the original article was complaining about. I think he was frustrated that while the tools were very good, they were hard to teach to his students. He wishes there were easier to learn systems for doing this work. Personally, I don't think it will ever happen because the problem space is quite complicated. You can make it easier to learn by restricting what it can do, but eventually you butt up against that ceiling, making it extremely difficult to use. That's why we are all still using these same tools decades years later (although, I'm quite happy to ditch make in favour of more modern equivalents [1]) I can't tell you the number of times I've wandered into a new project and spent every spare second writing README files, fixing the build systems, removing dead code, etc. I've even worked on projects where there wasn't a single person who knew how to build the entire project -- all they could do was compile individual files and link it to pre-built libraries. They thought me crazy for even wanting to know how to build it. We do our industry a grave disservice by not teaching this stuff somewhere in our education system. [1] Says the guy who just wrote a new make file a couple of weeks ago <whistling>