7 ms·

For those who don't recall: Back in the Dark Ages, there was a race to decode the human genome. The leading competitiors (wealthiest) were Celera Genomics and

by 10g1k 3y ago

For those who don't recall: Back in the Dark Ages, there was a race to decode the human genome. The leading competitiors (wealthiest) were Celera Genomics and the Human Genome Project. After some time, Celera (headed by Craig Venter), announced they had done the deed. However, what Celera had actually done was used what they called a "shotgun method", which meant they took small samples here and there, then built a model of the genome with various statistical shenanigans. About five years later, after all the hype, they admitted they had not sequenced the entire genome. Ref: https://www.technologyreview.com/2007/09/04/223919/craig-venters-genome/ https://www.technologyreview.com/2007/09/04/223919/craig-ven...

ekianjo 3y ago
[flagged]
- tomrod 3y ago
  I'm sorry, are you saying the media, whose job it is to report and not serve as experts, are useless ignorants, or Celera, which lied to the public, are useless idiots? If the first, time to let that dead dog lie. It's a tired trope.
  - kaliqt 3y ago
    Not a tired trope because that dog isn't dead. Until the media either 1. has no influence or 2. stops being dishonest then it needs to constantly be called out and berated.
    - lemmsjid 3y ago
      What if it is apparent that there are other groups pushing the narrative that the media is untrustworthy so that they themselves can own the narrative? Any given group of humans will have a mixture of deceptive and trustworthy participants. Singling one group out for constant castigation, held against an impossible standard, makes it easier for dishonest members of other groups to avoid the spotlight.
      - AnthonyMouse 3y ago
        "The media" isn't a uniform entity, but when people refer to it in this context they typically mean the large consolidated media corporations that own major print and television outlets. And those entities have essentially eroded their own credibility to the point that trusting anything they say is foolish. If they say the sky is blue, go out and check. They could do better, but until they do, it makes sense to castigate them when they err, because avoiding that reputational harm is their incentive to do better. Or, if they fail to do better, the thing that should ultimately kill them so they can be replaced with something that can do better.
        lemmsjid 3y ago
        The larger news outlets are some of the few remaining places that actually hold their news rooms to some kind of factual standard and facsimile of objective reality. If NYT or the Economist or the BBC tells me the sky is blue, I'm pretty sure it is. If they report on some scientific topic the reporter is not an expert on, I understand that there's probably some signal lost. If they publish something on a contested political topic, I understand there might be some slant in the direction of their overall political leaning. I also understand that they have an interest in the direction of their corporate ownership and host country. But there are plenty of other outlets trying (and succeeding) at capturing audiences who have all those same conflicts of interest, and more, but do not hold themselves to a factual standard. While media companies have indeed "eroded their own credibility", in that there's a steady stream of falseness alongside truth, I would add with equal measure that "their credibility has been undermined by others". The loudest voices that castigate the media, in America at least, tend to be the ones that profit the most from being the sole arbiter of truth for their own audiences. The business model of having an autonomous news room that holds itself to factual standards and is largely sequestered from advertising is shrinking and shrinking. I mourn its death, for I see no replacement coming along.
        ekianjo 3y ago
        > If NYT or the Economist or the BBC tells me the sky is blue Since these journalists are spending their time on Twitter and never put a foot outside on the real world, no, I dont trust anything they say.
    - johnnyanmac 3y ago
      I think Hanlon's razor applies here. ignorance is not dishonesty.
  - 000ooo000 3y ago
    I'd rather these ideas spread to others, potentially younger readers, than never have to read a trope again. I'm not sure it will ever change if we just accept it and never discuss it again.
- dekhn 3y ago
  In a sibling comment I give about 6 articles published over 3 decades by a single outlet- the Old Grey Lady (NY Times)- and it the competition was widely covered at the time.
dekhn 3y ago
Yes, but the HGP didn't make a full sequence either, using their scaffold-based contig assembly. Both groups declared a truce and announced they were "finished with the first draft" 35 years ago: https://www.nytimes.com/1987/12/13/magazine/the-genome-project.html https://www.nytimes.com/1987/12/13/magazine/the-genome-proje... 33 years ago: https://www.nytimes.com/1990/06/05/science/great-15-year-project-to-decipher-genes-stirs-opposition.html https://www.nytimes.com/1990/06/05/science/great-15-year-pro... 29 years ago: the competition gets fierce https://www.nytimes.com/1994/02/22/science/scientist-at-work-j-craig-venter-a-bold-short-cut-to-human-genes.html https://www.nytimes.com/1994/02/22/science/scientist-at-work... 24 years ago: the race is alive! https://www.nytimes.com/1999/03/23/science/who-ll-sequence-human-genome-first-it-s-up-to-phred.html https://www.nytimes.com/1999/03/23/science/who-ll-sequence-h... 23 years ago: draft is complete https://archive.nytimes.com/www.nytimes.com/library/national/science/062600sci-human-genome.html https://archive.nytimes.com/www.nytimes.com/library/national... 20 years ago: https://www.nytimes.com/2003/04/15/science/once-again-scientists-say-human-genome-is-complete.html https://www.nytimes.com/2003/04/15/science/once-again-scient... 2 years ago: https://archive.nytimes.com/www.nytimes.com/library/national/science/062600sci-human-genome.html https://archive.nytimes.com/www.nytimes.com/library/national... The celera approach using shotgun was tricky because assembling the bits required a great deal of computational finesse and horsepower; "The final assembly computations were run on Compaq’s new AlphaServer GS160 because the algorithms and data required 64 gigabytes of shared memory to run successfully." At the time the GS160 was the monster machine, a classic big iron UNIX, which could be tightly clustered- unified IO/filesystem between all the machines. The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST, and invented the Suffix Array with Udi Manber. The public project had its own issues, as many of the teams assigned to work on it still had the "cottage industry/artisinal/academic" approach, then Eric Lander came along and turned it into an industrial process, and parlayed that into running the Broad Institute, a privately funded MIT/Harvard research institute in Boston. These days petabytes of sequence data are generated every day and stored in clouds. The genome has been a fundamental tool for shaping our studies of humans, although its true potential for understanding complex phenotypes remains elusive.
- sublinear 3y ago
  > The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST... So... um... BLAST processing?
jhbadger 3y ago
As noted, the shotgun sequencing wasn't any worse than the traditional chromosome walking method used by the other human genome project, and neither created a truly complete sequence, which is only possible today with improvements in technology. And it is worth noting that despite all the critiques of shotgun sequencing at the time as "cheating", chromosome walking sequencing is dead, and shotgun is what we use today. If anything replaces that it will be long read technology like that done by Oxford Nanopore sequencing, although we aren't really there yet (you still need to assemble that data and don't really get end-to-end sequences yet).
- kazinator 3y ago
  I feel a joke coming on. Mother: "Amy, dear, you hardly touched any of your vegetables. Most of them are still on your plate." Amy: "That's not true, Mom; I completely sequenced them into my tummy using the shotgun method."
  - udkl 3y ago
    Someone submit this to smbc comics !
    - SebJansen 3y ago
      yes! or XKCD
- robwwilliams 3y ago
  Not true now with PacBio Revio and new Oxford Nanopore chemistry and flow cells. We still use a bit of older Illumina data for polishing, but long read methods now as accurate. HiC and Bionano more important as supplements than 150 nucleotide pair-end reads. And the cost of a T2T using a combination of these technologies is already well under $10,000 in reagent costs. The assembly is still a complex art especially for a messy chromosome like Y.
wslh 3y ago
TIL that. I was 100% sure the DNA was completely sequenced. Even have a friend with a sequencing company. It is incredible how ignorant one can be. Thank you 10g1k and HN.
- alexvitkov 3y ago
  Thought so too, turns out we're right now
  - throwawaymaths 3y ago
    I'm not really sure what it even means to be 100% sequenced. Does that mean that every possible polymorphism is accounted for? Does that mean that 100% of possible polymorphic sites are known? What about weird edge cases. You have a tandem repeat that is known, but in some small population of humans, there's two tandem repeats with an island of something else in there.
- AlecSchueler 3y ago
  Found out you were wrong on the day you became right!
- mnw21cam 3y ago
  FWIW, the DNA was 100% sequenced. However, a proportion of that data that came back from the sequencing machine was too similar to each other, so it was impossible to assemble it into a coherent full set of contiguous chromosomes. Imagine it like putting together a jigsaw puzzle, except that the picture on the front of the puzzle has lots of repeating motifs, so you end up with multiple pieces that look identical. You won't be able to tell how the whole thing fits together, but you can assemble the bits that are well-behaved and unique. Modern technology gives us larger jigsaw pieces, which allows us to distinguish between almost-identical parts of the puzzle better. But I would note that the project linked did a huge amount of sequencing using very expensive methods to be able to resolve the whole thing.
  - nzach 3y ago
    For anyone interested, the problem mostly boils down to the sequence lenght we can read. Modern sequencing techniques allow us o read 200 to 500 bases at a time. So after hat we need do find a way o arrange these short sequences into a single sequence. And this can be prety hard, especially when you are doing 'de novo' assembly[0]. Besides that, there is the fact that some regions of DNA are repeated[1]. [0] - https://en.wikipedia.org/wiki/Third-generation_sequencing#De_novo_assembly https://en.wikipedia.org/wiki/Third-generation_sequencing#De... [1] - https://en.wikipedia.org/wiki/Repeated_sequence_(DNA) https://en.wikipedia.org/wiki/Repeated_sequence_(DNA)
comboy 3y ago
Anybody familiar with news sources which questioned Celera accomplishment before they admitted how they did it?
- cowsandmilk 3y ago
  Parent is weird in claiming celera was hiding things. The second half of the 90s was a very vocal spat between Celera and the HGP about sequencing methods and Celera saying shotgun sequencing was superior. And the HGP was publishing updates to their genome every 24 hours while Celera published their sequence on completion of the initial draft, which showed their coverage. The fact that there wasn’t 100% coverage wasn’t hidden to anyone who read the published papers. (The abstract for the Celera paper even states “reduced the size and number of gaps”, which plainly indicates there were still gaps…)
  - comboy 3y ago
    Huh, narration matters. Thanks.
- throwawaymaths 3y ago
  http://news.bbc.co.uk/2/hi/science/nature/2940601.stm http://news.bbc.co.uk/2/hi/science/nature/2940601.stm > The decoding is now close to 100% complete. The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with. OP is being silly. Nobody was fooling anyone.
delecti 3y ago
I don't think this was a secret. I remember reading an article (IIRC in Readers Digest) nearly ~20 years ago which specifically referred to it as a "shotgun method". I wasn't (and still am not) enough of an expert to have known that was a less authentic/accurate method, but at least at the time the tone was not that they were doing anything illicit.
- throwawaymaths 3y ago
  if you go over the maths on the shotgun method, not taking into account systematic difficulty in sequencing (like tandem repeats), the likelihood of incomplete coverage vanishes much more rapidly than you'd expect, and after about 5-10x coverage volume, coverage is basically linear in the total amount of sequencing.
mannykannot 3y ago
My recollection of the contentious issues at the time are that a) the Celera project needed the information being made public by the NIH/international Human Genome Project in order to assemble sequences, but was keeping secret its own results, and b) it was intent on acquiring IP rights before the information entered the public domain.
- greazy 3y ago
  Yes this is correct. Shotgun sequencing needs to be aligned to a reference which in this case was publicly free to access.
SubiculumCode 3y ago
This comment is needlessly negative. What they did had value and they did not hide what they did either, at least that wasn't my impression. The statistical "shenanigans" is better termed as an innovation.
jjtheblunt 3y ago
Shotgun sequencing could be better explained than "statistical shenanigans": imagine blasting a huge collection of identical long strings with a shotgun, each breaking into different fragments, and realizing overlaps in the fragments, collectively, can be enough to uniquely restore one copy of the original string.
throwawaymaths 3y ago
Pretty sure that the announcement was that the sequence was "essentially complete" (or something like that). Here's a press report from the time that provided the caveat: http://news.bbc.co.uk/2/hi/science/nature/2940601.stm http://news.bbc.co.uk/2/hi/science/nature/2940601.stm > The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with.

by 10g1k 3y ago

ekianjo 3y ago

[flagged]

tomrod 3y ago
I'm sorry, are you saying the media, whose job it is to report and not serve as experts, are useless ignorants, or Celera, which lied to the public, are useless idiots? If the first, time to let that dead dog lie. It's a tired trope.
- kaliqt 3y ago
  Not a tired trope because that dog isn't dead. Until the media either 1. has no influence or 2. stops being dishonest then it needs to constantly be called out and berated.
  - lemmsjid 3y ago
    What if it is apparent that there are other groups pushing the narrative that the media is untrustworthy so that they themselves can own the narrative? Any given group of humans will have a mixture of deceptive and trustworthy participants. Singling one group out for constant castigation, held against an impossible standard, makes it easier for dishonest members of other groups to avoid the spotlight.
    - AnthonyMouse 3y ago
      "The media" isn't a uniform entity, but when people refer to it in this context they typically mean the large consolidated media corporations that own major print and television outlets. And those entities have essentially eroded their own credibility to the point that trusting anything they say is foolish. If they say the sky is blue, go out and check. They could do better, but until they do, it makes sense to castigate them when they err, because avoiding that reputational harm is their incentive to do better. Or, if they fail to do better, the thing that should ultimately kill them so they can be replaced with something that can do better.
      - lemmsjid 3y ago
        The larger news outlets are some of the few remaining places that actually hold their news rooms to some kind of factual standard and facsimile of objective reality. If NYT or the Economist or the BBC tells me the sky is blue, I'm pretty sure it is. If they report on some scientific topic the reporter is not an expert on, I understand that there's probably some signal lost. If they publish something on a contested political topic, I understand there might be some slant in the direction of their overall political leaning. I also understand that they have an interest in the direction of their corporate ownership and host country. But there are plenty of other outlets trying (and succeeding) at capturing audiences who have all those same conflicts of interest, and more, but do not hold themselves to a factual standard. While media companies have indeed "eroded their own credibility", in that there's a steady stream of falseness alongside truth, I would add with equal measure that "their credibility has been undermined by others". The loudest voices that castigate the media, in America at least, tend to be the ones that profit the most from being the sole arbiter of truth for their own audiences. The business model of having an autonomous news room that holds itself to factual standards and is largely sequestered from advertising is shrinking and shrinking. I mourn its death, for I see no replacement coming along.
        ekianjo 3y ago
        > If NYT or the Economist or the BBC tells me the sky is blue Since these journalists are spending their time on Twitter and never put a foot outside on the real world, no, I dont trust anything they say.
  - johnnyanmac 3y ago
    I think Hanlon's razor applies here. ignorance is not dishonesty.
- 000ooo000 3y ago
  I'd rather these ideas spread to others, potentially younger readers, than never have to read a trope again. I'm not sure it will ever change if we just accept it and never discuss it again.
dekhn 3y ago
In a sibling comment I give about 6 articles published over 3 decades by a single outlet- the Old Grey Lady (NY Times)- and it the competition was widely covered at the time.

dekhn 3y ago

Yes, but the HGP didn't make a full sequence either, using their scaffold-based contig assembly. Both groups declared a truce and announced they were "finished with the first draft" 35 years ago: https://www.nytimes.com/1987/12/13/magazine/the-genome-project.html https://www.nytimes.com/1987/12/13/magazine/the-genome-proje... 33 years ago: https://www.nytimes.com/1990/06/05/science/great-15-year-project-to-decipher-genes-stirs-opposition.html https://www.nytimes.com/1990/06/05/science/great-15-year-pro... 29 years ago: the competition gets fierce https://www.nytimes.com/1994/02/22/science/scientist-at-work-j-craig-venter-a-bold-short-cut-to-human-genes.html https://www.nytimes.com/1994/02/22/science/scientist-at-work... 24 years ago: the race is alive! https://www.nytimes.com/1999/03/23/science/who-ll-sequence-human-genome-first-it-s-up-to-phred.html https://www.nytimes.com/1999/03/23/science/who-ll-sequence-h... 23 years ago: draft is complete https://archive.nytimes.com/www.nytimes.com/library/national/science/062600sci-human-genome.html https://archive.nytimes.com/www.nytimes.com/library/national... 20 years ago: https://www.nytimes.com/2003/04/15/science/once-again-scientists-say-human-genome-is-complete.html https://www.nytimes.com/2003/04/15/science/once-again-scient... 2 years ago: https://archive.nytimes.com/www.nytimes.com/library/national/science/062600sci-human-genome.html https://archive.nytimes.com/www.nytimes.com/library/national... The celera approach using shotgun was tricky because assembling the bits required a great deal of computational finesse and horsepower; "The final assembly computations were run on Compaq’s new AlphaServer GS160 because the algorithms and data required 64 gigabytes of shared memory to run successfully." At the time the GS160 was the monster machine, a classic big iron UNIX, which could be tightly clustered- unified IO/filesystem between all the machines. The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST, and invented the Suffix Array with Udi Manber. The public project had its own issues, as many of the teams assigned to work on it still had the "cottage industry/artisinal/academic" approach, then Eric Lander came along and turned it into an industrial process, and parlayed that into running the Broad Institute, a privately funded MIT/Harvard research institute in Boston. These days petabytes of sequence data are generated every day and stored in clouds. The genome has been a fundamental tool for shaping our studies of humans, although its true potential for understanding complex phenotypes remains elusive.

sublinear 3y ago
> The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST... So... um... BLAST processing?

jhbadger 3y ago

As noted, the shotgun sequencing wasn't any worse than the traditional chromosome walking method used by the other human genome project, and neither created a truly complete sequence, which is only possible today with improvements in technology. And it is worth noting that despite all the critiques of shotgun sequencing at the time as "cheating", chromosome walking sequencing is dead, and shotgun is what we use today. If anything replaces that it will be long read technology like that done by Oxford Nanopore sequencing, although we aren't really there yet (you still need to assemble that data and don't really get end-to-end sequences yet).

kazinator 3y ago
I feel a joke coming on. Mother: "Amy, dear, you hardly touched any of your vegetables. Most of them are still on your plate." Amy: "That's not true, Mom; I completely sequenced them into my tummy using the shotgun method."
- udkl 3y ago
  Someone submit this to smbc comics !
  - SebJansen 3y ago
    yes! or XKCD
robwwilliams 3y ago
Not true now with PacBio Revio and new Oxford Nanopore chemistry and flow cells. We still use a bit of older Illumina data for polishing, but long read methods now as accurate. HiC and Bionano more important as supplements than 150 nucleotide pair-end reads. And the cost of a T2T using a combination of these technologies is already well under $10,000 in reagent costs. The assembly is still a complex art especially for a messy chromosome like Y.

wslh 3y ago

TIL that. I was 100% sure the DNA was completely sequenced. Even have a friend with a sequencing company. It is incredible how ignorant one can be. Thank you 10g1k and HN.

alexvitkov 3y ago
Thought so too, turns out we're right now
- throwawaymaths 3y ago
  I'm not really sure what it even means to be 100% sequenced. Does that mean that every possible polymorphism is accounted for? Does that mean that 100% of possible polymorphic sites are known? What about weird edge cases. You have a tandem repeat that is known, but in some small population of humans, there's two tandem repeats with an island of something else in there.
AlecSchueler 3y ago
Found out you were wrong on the day you became right!
mnw21cam 3y ago
FWIW, the DNA was 100% sequenced. However, a proportion of that data that came back from the sequencing machine was too similar to each other, so it was impossible to assemble it into a coherent full set of contiguous chromosomes. Imagine it like putting together a jigsaw puzzle, except that the picture on the front of the puzzle has lots of repeating motifs, so you end up with multiple pieces that look identical. You won't be able to tell how the whole thing fits together, but you can assemble the bits that are well-behaved and unique. Modern technology gives us larger jigsaw pieces, which allows us to distinguish between almost-identical parts of the puzzle better. But I would note that the project linked did a huge amount of sequencing using very expensive methods to be able to resolve the whole thing.
- nzach 3y ago
  For anyone interested, the problem mostly boils down to the sequence lenght we can read. Modern sequencing techniques allow us o read 200 to 500 bases at a time. So after hat we need do find a way o arrange these short sequences into a single sequence. And this can be prety hard, especially when you are doing 'de novo' assembly[0]. Besides that, there is the fact that some regions of DNA are repeated[1]. [0] - https://en.wikipedia.org/wiki/Third-generation_sequencing#De_novo_assembly https://en.wikipedia.org/wiki/Third-generation_sequencing#De... [1] - https://en.wikipedia.org/wiki/Repeated_sequence_(DNA) https://en.wikipedia.org/wiki/Repeated_sequence_(DNA)

comboy 3y ago

Anybody familiar with news sources which questioned Celera accomplishment before they admitted how they did it?

cowsandmilk 3y ago
Parent is weird in claiming celera was hiding things. The second half of the 90s was a very vocal spat between Celera and the HGP about sequencing methods and Celera saying shotgun sequencing was superior. And the HGP was publishing updates to their genome every 24 hours while Celera published their sequence on completion of the initial draft, which showed their coverage. The fact that there wasn’t 100% coverage wasn’t hidden to anyone who read the published papers. (The abstract for the Celera paper even states “reduced the size and number of gaps”, which plainly indicates there were still gaps…)
- comboy 3y ago
  Huh, narration matters. Thanks.
throwawaymaths 3y ago
http://news.bbc.co.uk/2/hi/science/nature/2940601.stm http://news.bbc.co.uk/2/hi/science/nature/2940601.stm > The decoding is now close to 100% complete. The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with. OP is being silly. Nobody was fooling anyone.

delecti 3y ago

I don't think this was a secret. I remember reading an article (IIRC in Readers Digest) nearly ~20 years ago which specifically referred to it as a "shotgun method". I wasn't (and still am not) enough of an expert to have known that was a less authentic/accurate method, but at least at the time the tone was not that they were doing anything illicit.

throwawaymaths 3y ago
if you go over the maths on the shotgun method, not taking into account systematic difficulty in sequencing (like tandem repeats), the likelihood of incomplete coverage vanishes much more rapidly than you'd expect, and after about 5-10x coverage volume, coverage is basically linear in the total amount of sequencing.

mannykannot 3y ago

My recollection of the contentious issues at the time are that a) the Celera project needed the information being made public by the NIH/international Human Genome Project in order to assemble sequences, but was keeping secret its own results, and b) it was intent on acquiring IP rights before the information entered the public domain.

greazy 3y ago
Yes this is correct. Shotgun sequencing needs to be aligned to a reference which in this case was publicly free to access.

SubiculumCode 3y ago

This comment is needlessly negative. What they did had value and they did not hide what they did either, at least that wasn't my impression. The statistical "shenanigans" is better termed as an innovation.

jjtheblunt 3y ago

Shotgun sequencing could be better explained than "statistical shenanigans": imagine blasting a huge collection of identical long strings with a shotgun, each breaking into different fragments, and realizing overlaps in the fragments, collectively, can be enough to uniquely restore one copy of the original string.

throwawaymaths 3y ago

Pretty sure that the announcement was that the sequence was "essentially complete" (or something like that). Here's a press report from the time that provided the caveat: http://news.bbc.co.uk/2/hi/science/nature/2940601.stm http://news.bbc.co.uk/2/hi/science/nature/2940601.stm > The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with.