4 ms·

Responding to criticism of the Immunity Project's lack of peer review: "If the traditional science community wants to say these things aren’t going to work, th

Responding to criticism of the Immunity Project's lack of peer review: "If the traditional science community wants to say these things aren’t going to work, they can keep doing that. That’s what used to happen with software companies. The Immunity Project may fail, and the next 30 things we fund may fail — and the whole model is such that that’s OK. We can have 31 failures and one success, and the model still works great. That’s how we’re designed."

pdq 13y ago
Unfortunately that's more of a defense of YC's funding of the Immunity Project than a defense of the Immunity Project itself.
- the_watcher 13y ago
  The point is they don't need to defend the Immunity Project. They are giving them the resources to (ideally) allow them to defend themselves. If they fail, YC is fine with that, but the Immunity Project at least has the chance to try something.
  - Fomite 13y ago
    I think a lot of the pushback, and absolutely my source for concern, was not so much YC funding the Immunity Project, but the hype spike that comes from a high profile investment around something with very, very little actual scientific evidence behind it.
tdaltonc 13y ago
I was more struck by this part: > How many Nature papers are not reproducible? To call out a new bunch of people who are trying something different because people are afraid they’ll fail is sort of hypocritical given that we don’t really check whether anybody else’s results are correct, either. Peer review is more about politics then reproducibility. Reproducibility is really important, but Nature doesn't seem to agree.
- ihnorton 13y ago
  This comes up a lot, and there is truth to it. But my flippant response is usually "the ones that aren't that important". In just the past two months we've seen examples of the system working well for both reproduction (CRISPR) and invalidation (acid bath-induced pluripotency) of spectacular results.
  - tdaltonc 13y ago
    Those are victories, but I wouldn't credit them to peer review. I think of "peer review" as what happens between a manuscript being submitted and it being published.
- collyw 13y ago
  I have a theory with peer review, if it works, it is important and it is reproducible, then people will reproduce it and build on it. If any of the three parts are not there, then no one will want / need / be able to reproduce it, and it will fade into scientific obscurity, while the rest of the scientific community keep moving forwards.
  - omnisci 13y ago
    Or people spend a lot of time and money trying to chase down something that a ground "found" and published. It's very common to read a paper, think it sounds good, then try build on it only to find that you can't. Peer review is completely flawed. Until it's made more transparent, like at F1000, it's going to continue that way. Not being a negative Nancy here, but I've been on both sides of the fence (reviewed and had grants/papers reviewed) and it isn't an efficient system. Much like many other things in science, it's an antiquated process that needs to be overhauled to get back to it's "roots".
    - collyw 13y ago
      Peer review is not perfect, but its still better than the way the mainstream media reports the news.
fabian2k 13y ago
The lack of any peer-reviewed papers backing up their approach is not a problem at all if professionals invest in that company. They should be able to get the data from the company and form their own opinion on it. It's a bit different if you crowdsource the whole thing. The individual backers don't have the ability and knowledge to dig as deep to actually judge the validity of the approach. So I can certainly understand that this draws some criticism.

pdq 13y ago

Unfortunately that's more of a defense of YC's funding of the Immunity Project than a defense of the Immunity Project itself.

the_watcher 13y ago
The point is they don't need to defend the Immunity Project. They are giving them the resources to (ideally) allow them to defend themselves. If they fail, YC is fine with that, but the Immunity Project at least has the chance to try something.
- Fomite 13y ago
  I think a lot of the pushback, and absolutely my source for concern, was not so much YC funding the Immunity Project, but the hype spike that comes from a high profile investment around something with very, very little actual scientific evidence behind it.

tdaltonc 13y ago

I was more struck by this part: > How many Nature papers are not reproducible? To call out a new bunch of people who are trying something different because people are afraid they’ll fail is sort of hypocritical given that we don’t really check whether anybody else’s results are correct, either. Peer review is more about politics then reproducibility. Reproducibility is really important, but Nature doesn't seem to agree.

ihnorton 13y ago
This comes up a lot, and there is truth to it. But my flippant response is usually "the ones that aren't that important". In just the past two months we've seen examples of the system working well for both reproduction (CRISPR) and invalidation (acid bath-induced pluripotency) of spectacular results.
- tdaltonc 13y ago
  Those are victories, but I wouldn't credit them to peer review. I think of "peer review" as what happens between a manuscript being submitted and it being published.
collyw 13y ago
I have a theory with peer review, if it works, it is important and it is reproducible, then people will reproduce it and build on it. If any of the three parts are not there, then no one will want / need / be able to reproduce it, and it will fade into scientific obscurity, while the rest of the scientific community keep moving forwards.
- omnisci 13y ago
  Or people spend a lot of time and money trying to chase down something that a ground "found" and published. It's very common to read a paper, think it sounds good, then try build on it only to find that you can't. Peer review is completely flawed. Until it's made more transparent, like at F1000, it's going to continue that way. Not being a negative Nancy here, but I've been on both sides of the fence (reviewed and had grants/papers reviewed) and it isn't an efficient system. Much like many other things in science, it's an antiquated process that needs to be overhauled to get back to it's "roots".
  - collyw 13y ago
    Peer review is not perfect, but its still better than the way the mainstream media reports the news.

fabian2k 13y ago

The lack of any peer-reviewed papers backing up their approach is not a problem at all if professionals invest in that company. They should be able to get the data from the company and form their own opinion on it. It's a bit different if you crowdsource the whole thing. The individual backers don't have the ability and knowledge to dig as deep to actually judge the validity of the approach. So I can certainly understand that this draws some criticism.