7 ms·
Amazon Genomics CLI
- bradfitz 5y agoc'mon, they couldn't have named it the Amazon Genomics Toolkit CLI or something to get a T in there and then made the binary be named "agtc" instead of "agc"?
- adreamingsoul 5y agoAWS prefers at most three worded service names. Three letter acronyms are easier to remember and pronounce than four.
- apaprocki 5y agoIs there data behind that claim? The Bloomberg Terminal uses up to four and they are not strictly acronyms — sure, WEI is “World Economic Indices”, but TOP is “Top News” and OMON is “Options Monitor”. Four opens up a much wider range of possibilities in English if you don’t strictly use acronyms because there’s a wealth of short words and abbreviations that are super memorable.
- aaauaucuggaa 5y agoa few examples that come to mind: EC2, EMR, ECR, EKS, RDS, EBS, DMS, ECS, VPC, SSS (S3), SNS, SES, IAM,
- high_byte 5y agoalthough in that case agtc is the obvious choice
- arduinomancer 5y agoAmazon Quantum Ledger Database (QLDB)
- pcrh 5y agoMaybe they thought agc was a more peaceful, serine, choice?
- vixen99 5y agoYou mean serene not the amino acid serine.
- SomeHacker44 5y agoYou two, so good. I cannot tell which comment is more amusing. :)
- rflrob 5y agoFor the non molecular biologists out there, the 3-base codon AGC gets converted to the amino acid Serine. https://en.m.wikipedia.org/wiki/DNA_and_RNA_codon_tables https://en.m.wikipedia.org/wiki/DNA_and_RNA_codon_tables
- zmmmmm 5y agoIt's nice to see cloud vendors starting to come to where the action is in standardised / open source tool sets rather than trying to capture people into their own proprietary workflow tools which has been a big turnoff for me thus far. I'd be curious if the underpinnings of the support they have for these tools allows more toolsets to be brought over or if the integration is highly specialised.
- geoffjentry 5y agoPeople underestimate how much compute genomics research chews up. The cloud companies (heck, even the CPU/GPU companies!) are for the most part trying to meet the researchers where they are instead of providing proprietary tools. For two reasons: a) They can sell a lot of compute time, storage space, etc b) It's an academic heavy field, and selling proprietary tools tend to be tough sledding
- epistasis 5y ago> the genomics and bioinformatics communities have developed specialized workflow definition languages like WDL, Nextflow, CWL, and Snakemake. It's funny, but after many years in the field, and many generations of workflow definitions, I'm still now happy with any of the options. Snakemake is the closest to being usable for me for both prototyping and longer term work does, but they are all still fragile and inflexible in weird ways. It's a tough problem to solve, which I think is evidenced by the large number of solutions. I've even tried looking outside at non-bioinformatics tools and never been very happy. And for all their flaws, the bioinformatics tools are better at typical bioinformatics workflows than tools developed for other domains.
- nuclearnice3 5y agoIt seems the domain specific aspect is important. When I read that sentence, I was surprised not to see the classic suggestions from tech/data science: Prefect, Airflow, Luigi, Apache NiFi, Jenkins, AWS Step Functions, or Pachyderm
- MontyCarloHall 5y agoThe reason they’re domain specific is that they were intended to be usable by non-programmers, namely wetlab biologists. Of course, that didn’t end up being the case. If you can’t program at an elementary enough level to define simple workflow graphs using an API in a non-DSL (like those you mention), then the language is irrelevant—you’re just not technically sophisticated enough to be writing your own workflows. So now we have these DSLs that non-programmers still can’t use and programmers all hate. A lose-lose.
- geoffjentry 5y agoThat's a big part of it. And I say this as someone who played a heavy role in the development of one of the named DSLs. Yes we had that exact intention and exact unintended outcome. There's another part of it as well. Bioinformatics workflows tend to be just enough different from more standard workflows that there is friction using off the shelf tooling. For one, the DAG nodes are often mostly/fully represented by command line tools expecting a POSIX style file system and making assumptions/asserting opinions on where files live, to where it writes outputs, etc. Bioinformatics workflow orchestrators can understand this and provide optimizations in the DSL to express how to manage the file movement. In contrast, I find many people with a more standard data/biz workflow mentality think of DAG nodes as being queries, running blocks of code, etc. Lifecycles vary as well. The institutions who have these workflows will either be pure research or some degree of research/production hybrid. There's an advantage to being able to use the same software on both side of that hybrid. When your research workflow is ready to be blessed to production, having to translate it into a different system can be expensive in terms of both time and bugs. Another aspect is that these workflows, until not too many years ago, were often run on HPC compute clusters using software like SGE, SLURM, PBS, etc. These DSLs can provide optimizations for tweaking parameters in a way that more mainstream tools do not.
- pixel_tracing 5y agoNo way in hell I’m giving my genomic data through CLI to Amazon
- anshumankmr 5y agoI am pretty sure this is not meant for an average consumer but for pharma companies and other life sciences based companies.
- vikramkr 5y agoI wonder how many of them would give that data to Amazon either. Thats some of the most valuable data a pharma company has
- geoffjentry 5y agoUse of Big Cloud is very common in this space, in particular AWS.
- anshumankmr 5y agoAmazon doesn't make websites, it gives you the tools to make one. Similarly, I don't think it would want to compete with pharma companies who have equally deep pockets and far more experience than Amazon would.
- geoffjentry 5y agoAlso academic research, but yes. These workflow orchestration tools are ubiquitous in the bioinformatics space. A lot mores than things like Airflow and the like.
- dekhn 5y agocould you articulate why? I ask, because I've shared my genome with Google, and it's online for anybody to see.
- fxtentacle 5y agoIndustry Espionage as a Service? In my old university, the people working on the genome studies had to sign a draconian NDA and they had a dedicated air-gapped compute cluster for it. If your genomics study can produce a patent for a new diagnostic method or maybe even a drug precursor molecule, that'll be way more valuable than everything you spent on security. And already 8 years ago, hacking attempts by what was thought of as foreign government actors was a common experience in life science research. When I had to give a talk about our methods, we scrambled all the gene IDs for fear of the slides getting leaked. So based on how people working in that field behave, I presume they would never ever agree to upload the data to servers outside of their control. Accordingly, I can only imagine this service to be used accidentally by inexperienced students (who are then on the hook for violating their NDA).
- rswail 5y agoIf you don't believe AWS's attestations about their security encryption and customer isolation and the associated certifications, then agreed, you shouldn't be sharing your data. That restricts you to your own infrastructure then. There are tradeoffs in both directions.
- IAmEveryone 5y agoI worked at a Max Planck genetics lab and never saw an NDA, or even any reference to the notion of keeping anything secret. The are also dozens of online databases for all sorts of data from biology. Reading between the lines, you seem to attach certain ideas to the term „genetics“?
- dekhn 5y agothey're talking about clinical trials with PII, and some associated genetic sequencing data.
- dekhn 5y agowhoever set up your system was overly paranoid. there are exabytes of PII in Amazon now. Air gapping was never a good security method.