3 ms·

After seeing all the fakery from how AI models handle programming (being "very good" at tasks where it turns out it was just trained on the problem in the first

by boxed 3y ago

After seeing all the fakery from how AI models handle programming (being "very good" at tasks where it turns out it was just trained on the problem in the first place), I do wonder if AlphaFold is likewise a red herring. How many of the AlphaFold protein folding predictions have been actually verified by a lab? And are any of those actually new proteins or just tiny variations on things in the training set?

hulium 3y ago
The linked article is not about AlphaFold itself, but about researchers verifying AlphaFold's predictions. So the article directly addresses your question. Unfortunately HN only shows the first half of the headline. > Shoichet agrees that AlphaFold predictions are not universally useful. “There were a lot of models that we didn’t even try because we thought they were so bad,” he says. But he estimates that in about one-third of cases, an AlphaFold structure could jump-start a project. “Compared to actually going out and getting a new structure, you could advance the project by a couple of years and that’s huge,” he says.
mechagodzilla 3y ago
Alphafold was largely trained on the structures in the Protein Data Bank, which includes lots of important protein targets. Just by virtue of evolutionary biology, many proteins are variants of one another, so if you're doing something like running AF on all of the proteins in the human body (only 24k or so), you're including lots of targets that were part of the training set, lots of targets that are closely related, and lots of targets we don't have structures for. Many (most?) proteins are also dynamic, so can't really be represented well by a single structure (rotating a sidechain inside a pocket when using a structure for virtual screening will frequently yield completely different 'hit' molecules, for instance). AF2 is a better guesser than previous tools for a reasonable structure, but that's like 1 step in a 47 step process to 'finding a new drug.' Even here, "AlphaFold" didn't find the drugs candidates, it generated the structure(s) that they ran common virtual screening tools on (which introduces all kinds of other assumptions and approximations depending on the tool).
Kiro 3y ago
I don't agree with your assessment of AI and programming at all. In fact, I think you're being insidious when framing your skepticism against this using words such as "likewise", making it sound like it has been unanimously agreed upon. It's like me saying "after seeing that the earth is flat, why wouldn't other planets be flat?". AI is definitely capable of solving novel programming problems and I'm constantly surprised how well Copilot understands the esoteric context of my code.
- nh23423fefe 3y ago
  This kind of formless skepticism is never well founded right? GP could've read the papers, or read commentary of the papers, or read about the 13th Critical Assessment of Structure Prediction (CASP) contest. Instead, "llms are bad, so isn't this bad too"
- boxed 3y ago
  > AI is definitely capable of solving novel programming problems I don't believe that. I think you are being fooled by the enormity of the training set, followed by the non-novelty of things you personally haven't seen before. I'm not saying AI tools aren't useful though. I'm just saying they are what they are: a fuzzy search in an enormous problem space. This is impressive and cool, but not the same as being able to solve truly novel problems.

by boxed 3y ago

hulium 3y ago

The linked article is not about AlphaFold itself, but about researchers verifying AlphaFold's predictions. So the article directly addresses your question. Unfortunately HN only shows the first half of the headline. > Shoichet agrees that AlphaFold predictions are not universally useful. “There were a lot of models that we didn’t even try because we thought they were so bad,” he says. But he estimates that in about one-third of cases, an AlphaFold structure could jump-start a project. “Compared to actually going out and getting a new structure, you could advance the project by a couple of years and that’s huge,” he says.

mechagodzilla 3y ago

Alphafold was largely trained on the structures in the Protein Data Bank, which includes lots of important protein targets. Just by virtue of evolutionary biology, many proteins are variants of one another, so if you're doing something like running AF on all of the proteins in the human body (only 24k or so), you're including lots of targets that were part of the training set, lots of targets that are closely related, and lots of targets we don't have structures for. Many (most?) proteins are also dynamic, so can't really be represented well by a single structure (rotating a sidechain inside a pocket when using a structure for virtual screening will frequently yield completely different 'hit' molecules, for instance). AF2 is a better guesser than previous tools for a reasonable structure, but that's like 1 step in a 47 step process to 'finding a new drug.' Even here, "AlphaFold" didn't find the drugs candidates, it generated the structure(s) that they ran common virtual screening tools on (which introduces all kinds of other assumptions and approximations depending on the tool).

Kiro 3y ago

I don't agree with your assessment of AI and programming at all. In fact, I think you're being insidious when framing your skepticism against this using words such as "likewise", making it sound like it has been unanimously agreed upon. It's like me saying "after seeing that the earth is flat, why wouldn't other planets be flat?". AI is definitely capable of solving novel programming problems and I'm constantly surprised how well Copilot understands the esoteric context of my code.

nh23423fefe 3y ago
This kind of formless skepticism is never well founded right? GP could've read the papers, or read commentary of the papers, or read about the 13th Critical Assessment of Structure Prediction (CASP) contest. Instead, "llms are bad, so isn't this bad too"
boxed 3y ago
> AI is definitely capable of solving novel programming problems I don't believe that. I think you are being fooled by the enormity of the training set, followed by the non-novelty of things you personally haven't seen before. I'm not saying AI tools aren't useful though. I'm just saying they are what they are: a fuzzy search in an enormous problem space. This is impressive and cool, but not the same as being able to solve truly novel problems.