3 ms·

The need for validation is itself something that needs to be validated. If models can make predictions that are true >99.99% of the time, validation may only be

The need for validation is itself something that needs to be validated. If models can make predictions that are true >99.99% of the time, validation may only be needed for situations where even 0.01% error rates are intolerable. In any case, even if some validation is needed, AI can speed up this kind of science by at least an order of magnitude.

Retric 2y ago
The question becomes how do you calculate that 99.99%. As soon as you use reserved training data to pick a model the score it got on that data is no longer a valid answer due to selection bias.
- trashtester 2y ago
  You validate a random selection of results using actual experiments. If you validate 10k results and a maximum of 1 of the validations contradicts the prediction, you're at about 99.99% accuracy. 10k experiments may seem like a lot, but keep in mind that if we can engineer nanobots out of proteins the same way we build engines from steel today, the number of "parts" we may want to build using such biological nanotech may easily go into the millions. And this kind of AI may very well be as useful for such tech as CAD is today. Or rather, it can be like the CAD + the engineer.
  - Retric 2y ago
    > using actual experiments That’s the bottleneck the model was trying to avoid in the first place. The goal of science is to come up with models we don’t need to validate before use, and it’s inherently iterative. Nanbots are more sci-fi magic than real world possible. In the real world we are stuck with things closer to highly specialized cellular machinery than some do anything grey goo. Growing buildings from local materials seems awesome until you realize just how slow trees grow and why.
    - trashtester 2y ago
      > That’s the bottleneck the model was trying to avoid in the first place. Some real world validation is always needed, but if the validations that are performed show high accuracy, the number of experiments will go down a lot. > Nanbots are more sci-fi magic than real world possible. Let's revisit this one in 10 years.
      - Retric 2y ago
        > revisit their one in 10 years The underlying physics isn’t changing in 100 years. Individual components can be nanoscale within controlled environments, but you simply need more atoms to operate independently.
achierius 2y ago
I think the problem is that it's not anywhere close to 99.99% -- I might be wrong but my understanding is that it's closer to 70%. So even if you can predict with perfect accuracy the cases that it gets wrong, you would still only cut down the validation cost by less than one OoM.
- crackalamoo 2y ago
  I'm not exactly sure how you would define 99.99% or 70% correct for protein folding, but AlphaFold is extremely accurate. The original AlphaFold paper had an rmsd of 0.96 Å at 95% residue coverage, which is less than the diameter of a carbon atom (1.4 Å).

Retric 2y ago

The question becomes how do you calculate that 99.99%. As soon as you use reserved training data to pick a model the score it got on that data is no longer a valid answer due to selection bias.

trashtester 2y ago
You validate a random selection of results using actual experiments. If you validate 10k results and a maximum of 1 of the validations contradicts the prediction, you're at about 99.99% accuracy. 10k experiments may seem like a lot, but keep in mind that if we can engineer nanobots out of proteins the same way we build engines from steel today, the number of "parts" we may want to build using such biological nanotech may easily go into the millions. And this kind of AI may very well be as useful for such tech as CAD is today. Or rather, it can be like the CAD + the engineer.
- Retric 2y ago
  > using actual experiments That’s the bottleneck the model was trying to avoid in the first place. The goal of science is to come up with models we don’t need to validate before use, and it’s inherently iterative. Nanbots are more sci-fi magic than real world possible. In the real world we are stuck with things closer to highly specialized cellular machinery than some do anything grey goo. Growing buildings from local materials seems awesome until you realize just how slow trees grow and why.
  - trashtester 2y ago
    > That’s the bottleneck the model was trying to avoid in the first place. Some real world validation is always needed, but if the validations that are performed show high accuracy, the number of experiments will go down a lot. > Nanbots are more sci-fi magic than real world possible. Let's revisit this one in 10 years.
    - Retric 2y ago
      > revisit their one in 10 years The underlying physics isn’t changing in 100 years. Individual components can be nanoscale within controlled environments, but you simply need more atoms to operate independently.

achierius 2y ago

I think the problem is that it's not anywhere close to 99.99% -- I might be wrong but my understanding is that it's closer to 70%. So even if you can predict with perfect accuracy the cases that it gets wrong, you would still only cut down the validation cost by less than one OoM.

crackalamoo 2y ago
I'm not exactly sure how you would define 99.99% or 70% correct for protein folding, but AlphaFold is extremely accurate. The original AlphaFold paper had an rmsd of 0.96 Å at 95% residue coverage, which is less than the diameter of a carbon atom (1.4 Å).