5 ms·
> Like most bioinformatics programs, AlphaFold 2 comes equipped with a “preprocessing pipeline”, which is the discipline’s lingo for “a Bash script that calls s
by londons_explore 5y ago
> Like most bioinformatics programs, AlphaFold 2 comes equipped with a “preprocessing pipeline”, which is the discipline’s lingo for “a Bash script that calls some other codes”.
Having Bioinformatics people requiring to stray a long way from their core competency to learn a scripting language from the 80's to write glue code seems... suboptimal. How many hours of expert time has been wasted figuring out how to split a string in bash?
Can us software people build a better tool to eliminate the need for this?
- johnnycerberus 5y agoI think the tools already exist just that people are conservative with their choices.
- ethbr0 5y ago(Obligatory xkcd; you know the one)
- FartyMcFarter 5y agoNot really?
- londons_explore 5y agohttps://xkcd.com/927/ https://xkcd.com/927/
- pbronez 5y agoMLops is a pretty hot area right now. The industry is trying to figure out how to engineer these things in a robust way. It’s not ubiquitous at all yet. There are lots of tools that wrap K8s and help you train models, but trying in DataOps in a robust way…. I haven’t seen the definitive answer yet. Seriously please tell me if you are founding this company so I can invest.
- cinntaile 5y agoPretty much anyone that works with data has to clean their data, learning to use the tools to do that is important. Whether that's bash, perl, R, python, ... doesn't really matter that much. If they already know bash then bash is a good tool since they can now focus on their data instead of wasting time on learning a new tool to do the same thing.
- culopatin 5y agoIt’s sad. Bio people really need friends in software to boost their research. Unfortunately they do a codeacademy python course for 5 min and try to get their projects going. Sometimes they succeed, sometimes they fail. But they don’t really have much time to dedicate to properly learn software dev and it’s not what they are into anyway, it’s a necessity. I think we could create something like a GitHub of bio projects that need help and people assist with hopes of getting their names in a paper
- ackbar03 5y agoI think DeepMind went one step further and solved the entire problem for them. They don't even have to touch their keyboards anymore
- stingraycharles 5y agoSure there are better alternatives, but the advantage of bash / shell scripting is that it’s very easy to glue a whole collection of tools together, and that expertise in this transfers well between domains. They probably could have achieved the same by invoking things in Python, but it would have been slower and not achieved a lot, other than “not using shell scripts”. And once you go down the path of optimizing this enough, you’ll end up reinventing shell scripts altogether.
- sanxiyn 5y agoWell, AlphaFold 2 generates MSA by invoking things in Python: https://github.com/deepmind/alphafold/blob/main/alphafold/data/tools/jackhmmer.py https://github.com/deepmind/alphafold/blob/main/alphafold/da.... So the article is actually mistaken on this point.
- knuthsat 5y agoFrom looking at the code, Bash looks pretty clean. I also use Bash and AWK for preprocessing a lot.
- laichzeit0 5y agoI used to be that guy as well, till a college convinced me that anything I can do in Bash, Awk I could probably do easier in Perl. Then everyone sort of drifted to Python. I get that if you never used Perl it’s pointless to learn it if you’re already in the Python stack, but… damn Perl’s regular expressions and how it’s so baked into the syntax of the language makes using regex in Python seem like going back to the Stone Age.
- volta83 5y agoBash is easier to explain and use than, eg, explaining ppl how to use Python subprocess Module Launch different apps, capture their output, etc. I find astonishing how bad Python is as a bash replacement. I often rather write an argument parser in bash than use Python if I have to invoke a bunch of commands.
- hortense 5y agoPython is bad, but bash is worse as soon as you need any kind of logic. > explaining ppl how to use Python subprocess Module Launch different apps, capture their output There's no shame in using `os.system`.
- maliker 5y agoWell, technically subprocess.check_call if you need to capture output.
- couteiral 5y agoThere is a growing trend to include Docker (or Singularity, which is more compatible with HPC architectures common in bioinformatics) images alongside codes. In particular, AlphaFold 2 does provide a Dockerfile, and they even include a Python "launcher script" hiding all the details of running the code. Sadly, this is very uncommon in the community. In a bioinformatics meeting, the sentence "I spent X days setting up Y software" will not raise many eyebrows
- maliker 5y agoI work in power systems, and the situation is similar. Maybe worse because paper authors often come up with new computational techniques but don’t implement them in code (much less code with a dockerfile). I look with much jealousy over at the computer science field where papers often include code, multiple versions under version control, automated tests, setup/docker scripts, and demonstration workflows and interfaces.
- dfas231 5y ago* Can us software people build a better tool to eliminate the need for this? Most probably, not. Bash is currently the sweet spot. It is actually the best tool for this job. Any other option comes with increased complexity, will make the whole software less stable.
- cabalamat 5y ago> How many hours of expert time has been wasted figuring out how to split a string in bash? Probably a good many more than would be needed to learn how to split a string in Python.
- 6gvONxR4sf7o 5y agoAmong the bioinformatics folks I know, bash is already a core competency. If you’re using your average biologist as your mental model, you’re thinking of the wrong people.
- jweather 5y agoI don't see this as a bad thing. Shell scripts are a great way to prototype text processing pipelines using multiple smaller components.