5 ms·

Lets not forget how Perl Saved The Human Genome Project [1], which I remember seeing in a print magazine at the time. As someone who is in the genomics world n

by jghn 3y ago

Lets not forget how Perl Saved The Human Genome Project [1], which I remember seeing in a print magazine at the time. As someone who is in the genomics world now as a software person, I find it amusing that I've gotten more into perl over the last year or so. It has its place, in the way that grep/sed/awk/etc does. [1] https://news.ycombinator.com/item?id=30327812 https://news.ycombinator.com/item?id=30327812

dekhn 3y ago
Lincoln Stein is great, and perl was certainly critical to many processes, but it was and remains fairly niche in genomics, which used much more C++, Java, and later Python. IMHO the person who "saved the genome project" was WJ Kent, who developed the assembler, BLAT, that the public project needed. I strive to point out that he wasn't a sole hero, nor was Lincoln. What I really like about BLAT is that while Celera was using Big Iron UNIX (massive 64-bit 64GB machines with 10s of terabytes of central high performance storage), BLAT ran on a cluster of linux machines, right around the time that people were waking up to the fact that linux was becoming a useful tool for scientific data processing. BLAT's design allowed it to work on a cluster of cheaper/smaller machines, while celera's algorithm really needed a massive shared memory single machine. it was sort of a microcosm of the larger battle being fought between Big iron UNIX and little intel linux at the time.
- jfoutz 3y ago
  I know hn isn't the place for memes, but, I'm obligated. Imagine what they could do with a beowulf cluster of pentium pros?
  - jghn 3y ago
    I feel like this is the one domain where the beowfulf cluster meme is still funny
  - epistasis 3y ago
    There was a NYTimes profile at the time of Kent's efforts, and they IIRC literally got the UC Santa Cruz administration to hand over desktop computers destined to go to various people for a few months to assemble into a cluster. So it was likely a bunch of pentium pros or similar. Not a Beowulf cluster, of course, but whatever clustering software Kent had hand rolled out of C for the purpose.
  - jacquesm 3y ago
    They weren't using pentium pros, they were using Alpha's because the memory interconnect between a beowulf cluster of pentium pros would be orders of magnitude slower. So likely 'the same, but much slower'.
    - jfoutz 3y ago
      oh, ha. yeah, that was sort of the root of the joke. companies like Celera, or whoever had the money would write a (large) check for big iron. smaller organizations were realizing they could get slower computers power pretty cheap, and they could get a whole lot of them. There was a little flurry of activity with flocking algorithms, objective-c had a little renaissance due to swarm computing. probably the most famous and lasting was map-reduce, from alphabet, but they were called google back then. there were a bunch of clever little tricks, like channel bonded network cards to make multiple cards look like one fast card, so you could double or triple bandwidth. The beowulf cluster joke was kind of the spirit of the scrappy, make something cool out of junk approach while calling companies like sun, hp, compaq dinosaurs. and it continued for a while with people building stuff like this - https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-playstation2-cluster/ https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-p... I think there was some weirdness with export controls of ps2's for this kind of stuff. I don't remember what pc chip was the new hotness back then, it was a good 20 years ago and my memory is dim. but that, I think, captures the gist of the meme.
      - jacquesm 3y ago
        www.clustercompute.com
- jghn 3y ago
  I agree from the human genome sequencing perspective and the superlatives of saving the HGP. But at the time I was a standard software dev who thought the topic was cool, as was this new fan dangled language named perl. However, I got into the genomics world in the early aughts, and the perl hung on and on and on. I remember starting a new job in the mid-teens, and one of the first tasks I had was to port over a legacy perl script. Such is the world of scientific software. 99% of the people I know who have touched perl since ~2003ish are in the bioinformatics space.
- iten 3y ago
  Small correction: BLAT is a local alignment tool Jim Kent also wrote. I think his assembler you're referring to is GigAssembler (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/).
  - heyoni 3y ago
    I also love that all of their genomic tools are released as a virtualbox virtual machine and their entire site can basically be run locally with personal genomic data. I think it’s called genome in a box. And if I remember correctly BLAT was so useful because it could be run on machines with less cpu power by loading more data into memory…or was it the other way around?
- ryandv 3y ago
  I had the privilege of working a co-op term at Stein's lab at the OICR. I encountered quite a bit of Perl during my short time there, and have yet to see it elsewhere in the modern enterprise tech world. BioPerl in particular stands out as a fairly substantial project in the bioinformatics space.
  - jboy55 3y ago
    Amazon still has some Perl kicking around, mainly on the main website. I don't think there's any new development though, just maintenance and gradual replacement.
- robwwilliams 3y ago
  And Jim Kent!
kapilkaisare 3y ago
Hi, I'm in software and looking to get into genomics/biotech. I'd like to learn more about how you made the move. Is there a way I could dm you? I saw no email in your profile.
freedomben 3y ago
Re: Perl, I very much agree. I've mostly switched to Ruby and `grep -P` but Perl really does have a place. `perl -pie` is still the most expressive/powerful option in some cases. Even if not directly used much anymore, I think most people vastly underappreciate how much Perl affected it's successors. Even as historical significance it's worthy of admiration.

by jghn 3y ago

dekhn 3y ago

Lincoln Stein is great, and perl was certainly critical to many processes, but it was and remains fairly niche in genomics, which used much more C++, Java, and later Python. IMHO the person who "saved the genome project" was WJ Kent, who developed the assembler, BLAT, that the public project needed. I strive to point out that he wasn't a sole hero, nor was Lincoln. What I really like about BLAT is that while Celera was using Big Iron UNIX (massive 64-bit 64GB machines with 10s of terabytes of central high performance storage), BLAT ran on a cluster of linux machines, right around the time that people were waking up to the fact that linux was becoming a useful tool for scientific data processing. BLAT's design allowed it to work on a cluster of cheaper/smaller machines, while celera's algorithm really needed a massive shared memory single machine. it was sort of a microcosm of the larger battle being fought between Big iron UNIX and little intel linux at the time.

jfoutz 3y ago
I know hn isn't the place for memes, but, I'm obligated. Imagine what they could do with a beowulf cluster of pentium pros?
- jghn 3y ago
  I feel like this is the one domain where the beowfulf cluster meme is still funny
- epistasis 3y ago
  There was a NYTimes profile at the time of Kent's efforts, and they IIRC literally got the UC Santa Cruz administration to hand over desktop computers destined to go to various people for a few months to assemble into a cluster. So it was likely a bunch of pentium pros or similar. Not a Beowulf cluster, of course, but whatever clustering software Kent had hand rolled out of C for the purpose.
- jacquesm 3y ago
  They weren't using pentium pros, they were using Alpha's because the memory interconnect between a beowulf cluster of pentium pros would be orders of magnitude slower. So likely 'the same, but much slower'.
  - jfoutz 3y ago
    oh, ha. yeah, that was sort of the root of the joke. companies like Celera, or whoever had the money would write a (large) check for big iron. smaller organizations were realizing they could get slower computers power pretty cheap, and they could get a whole lot of them. There was a little flurry of activity with flocking algorithms, objective-c had a little renaissance due to swarm computing. probably the most famous and lasting was map-reduce, from alphabet, but they were called google back then. there were a bunch of clever little tricks, like channel bonded network cards to make multiple cards look like one fast card, so you could double or triple bandwidth. The beowulf cluster joke was kind of the spirit of the scrappy, make something cool out of junk approach while calling companies like sun, hp, compaq dinosaurs. and it continued for a while with people building stuff like this - https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-playstation2-cluster/ https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-p... I think there was some weirdness with export controls of ps2's for this kind of stuff. I don't remember what pc chip was the new hotness back then, it was a good 20 years ago and my memory is dim. but that, I think, captures the gist of the meme.
    - jacquesm 3y ago
      www.clustercompute.com
jghn 3y ago
I agree from the human genome sequencing perspective and the superlatives of saving the HGP. But at the time I was a standard software dev who thought the topic was cool, as was this new fan dangled language named perl. However, I got into the genomics world in the early aughts, and the perl hung on and on and on. I remember starting a new job in the mid-teens, and one of the first tasks I had was to port over a legacy perl script. Such is the world of scientific software. 99% of the people I know who have touched perl since ~2003ish are in the bioinformatics space.
iten 3y ago
Small correction: BLAT is a local alignment tool Jim Kent also wrote. I think his assembler you're referring to is GigAssembler (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/).
- heyoni 3y ago
  I also love that all of their genomic tools are released as a virtualbox virtual machine and their entire site can basically be run locally with personal genomic data. I think it’s called genome in a box. And if I remember correctly BLAT was so useful because it could be run on machines with less cpu power by loading more data into memory…or was it the other way around?
ryandv 3y ago
I had the privilege of working a co-op term at Stein's lab at the OICR. I encountered quite a bit of Perl during my short time there, and have yet to see it elsewhere in the modern enterprise tech world. BioPerl in particular stands out as a fairly substantial project in the bioinformatics space.
- jboy55 3y ago
  Amazon still has some Perl kicking around, mainly on the main website. I don't think there's any new development though, just maintenance and gradual replacement.
robwwilliams 3y ago
And Jim Kent!

kapilkaisare 3y ago

Hi, I'm in software and looking to get into genomics/biotech. I'd like to learn more about how you made the move. Is there a way I could dm you? I saw no email in your profile.

freedomben 3y ago

Re: Perl, I very much agree. I've mostly switched to Ruby and `grep -P` but Perl really does have a place. `perl -pie` is still the most expressive/powerful option in some cases. Even if not directly used much anymore, I think most people vastly underappreciate how much Perl affected it's successors. Even as historical significance it's worthy of admiration.