5 ms·

Ex-Meta scientists debut gigantic AI protein design model

124 points by gmays 2y ago

trhway 2y ago
Tangential - the laws of nature discovered by our brain usually involve just few quantities, like f=ma. On one side it is a great ability of our brain for analytical reduction, on the other side it is just an inability to deal with complex multiparameter phenomena without such a reduction. I wonder if pumping more and more data into the NNs we'd be able to distill emerging multiparameter correlations which happen to be new laws of nature irreducible to more simpler ones.
- spywaregorilla 2y ago
  Eh, not in this context at least. Nature is a big term. Physics just "is". This makes it a good place for simple mechanics to appear and be derived. You can argue around how universal that is all you want but proteins themselves are the result of an enormously large search of a hugely high dimensional space. It doesn't seem possible for them to ever be reduced.
- ben_w 2y ago
  I disagree, our brains almost always deal with complex multiparameter phenomena. The part of us that can't cope with that is our System 2 reasoning: our minds' rules based thinking is great at F=ma but cannot enumerate the rules necessary at the level of retina cell activation patterns to recognise a tiger; our System 1 can, and always does, we just call it "intuition" or "common sense" or "a gut feeling". https://en.wikipedia.org/wiki/Dual_process_theory https://en.wikipedia.org/wiki/Dual_process_theory
- akreal 2y ago
  It took the modern human brain quite a bit of observations to discover and formulate f=ma, it was originally published in 1687.
btbuildem 2y ago
I wonder if this will ever come full circle (or.. spiral) and the AI tools we've created will in turn lead the way to discovering / inventing new proteins / cells / life forms that eventually outsmart and outcompete us.
- szundi 2y ago
  Isn’t the Terminator good enough?
  - warkdarrior 2y ago
    As usual, life is stranger than fiction.
yeutterg 2y ago
Just want to congratulate Tom Hayes on the big launch!
Spacecosmonaut 2y ago
Very nice work. We need brighter fluorescent protein tags that are more compact, in particular in the far red spectrum. The size of current fluorescent protein coding DNA sequences is out of reach of prime editing and still relies on less efficient gene editing technology.
- alan-hn 2y ago
  But they aren't brighter...
  - changoplatanero 2y ago
    its not more compact either, right?
  - Spacecosmonaut 2y ago
    There is only so much that you can get out of tinkering with naturally evolved proteins. I suspect that these kind of tools can be used to generate smaller & brighter FP's in the near future.
thebeardisred 2y ago
> For a smaller open-source version, certain sequences, such as those from viruses and a US government list of worrying pathogens and toxins, were excluded from training. Neither can ESM3-open — which scientists anywhere can download and run independently — be prompted to generate such proteins. That sounds like a glove being thrown down.
trott 2y ago
> matching less than 60% of the sequence of the most closely related fluorescent protein > When the researchers made around 100 of the resulting designs, several were as bright as natural GFPs, which are still vastly dimmer than lab-engineered variants. So they didn't come up with better functionality, unlike what some commentators imply. They basically introduced a bunch of mutations while preserving the overall function. Relevant: https://en.wikipedia.org/wiki/Conservative_replacement https://en.wikipedia.org/wiki/Conservative_replacement
- civilized 2y ago
  If the mutations were non-synonymous, resulting in different amino acids, the fact that they keep the natural function is still kinda cool. Very much a pure research result AFAICT, but worth a little something.
  - flobosg 2y ago
    This is already a well known fact: protein structure (and consequently function) is much more conserved than sequence, mostly due to biophysical constraints.
    - civilized 2y ago
      Forgive my ignorance - is that true even for non-synonymous mutations? They usually have a hard time disrupting the big structure?
      - flobosg 2y ago
        If non-synonymous mutations do not change the biophysical features of the amino acid residues, then the structure is usually kept. Alternatively, it can be the case that a disruptive mutation is compensated by another one that keeps the structure/function/phenotype. This is the basis for evolutionary coupling based structure prediction methods, such as Alphafold.
  - dekhn 2y ago
flobosg 2y ago
> However, its amino-acid sequence is vastly different, matching less than 60% of the sequence of the most closely related fluorescent protein in its training data set. Not to downplay this achievement, but 60% sequence identity is nowhere near “vastly different”.
- trott 2y ago
  The twilight zone is 20-35%: https://pubmed.ncbi.nlm.nih.gov/10195279/ https://pubmed.ncbi.nlm.nih.gov/10195279/ (Incidentally, the author was on my thesis committee, but this isn't precisely my field of expertise.)
  - flobosg 2y ago
    Proteins from the same family (and thus the same fold) can share less than 10% identity and still keep the same functionality as shown by e.g. profile Hidden Markov Model comparisons.
throwaway24124 2y ago
Are there any good resources for understanding models like this? Specifically a "protein language model". I have a basic grasp on how LLMs tokenize and encode natural language, but what does a protein language actually look like? An LLM can produce results that look correct but are actually incorrect, how are proteins produced by this model validated? Are the outputs run through some other software to determine whether the proteins are valid?
- cottonseed 2y ago
  Proteins are linear molecules consisting of sequences of (mostly) 20 amino acids. You can see the list of amino acids here: https://en.wikipedia.org/wiki/Amino_acid#Table_of_standard_amino_acid_abbreviations_and_properties https://en.wikipedia.org/wiki/Amino_acid#Table_of_standard_a.... There is a standard encoding of amino acids using single letters, A for alanine, etc. Earlier versions of ESM (I haven't read the ESM3 paper yet) uses one token per amino acid, plus a few control tokens (beginning of sequence, end of sequence, class token, mask, etc.) Earlier versions of ESM were BERT-style models focused on understanding, not GPT-style generative models.
- hn_throwaway_99 2y ago
  Agreed, would be interested if someone with more knowledge could comment. My layman's understanding of LLMs is that they are essentially "fancy autocomplete". That is, you take a whole corpus of text, then train the model to determine the statistical relationships between those words (more accurately, tokens), so that given a list of tokens of length N, the LLM will find the next most likely token for N + 1, and then to generate whole sentences/paragraphs, you just recursively repeat this process. I certainly understand encoding proteins as just a linear sequence of tokens representing their amino acids, but how does that then map to a human-language description of the function of those proteins?
  - changoplatanero 2y ago
    Most protein language models are not able to understand human-language descriptions of proteins. Mostly they just predict the next amino acid in a sequence and sometimes they can understand certain structured metadata tags.
brcmthrowaway 2y ago
Can protein models help cure prion diseases?
- Exoristos 2y ago
  At present this is unclear. But on the other hand, think of all the potential new prion diseases!
zacharyvoase 2y ago
Reading this it sounds like 'AI' is when you build a heuristic model (which we've had for a while now) but pass some threshold of cost in terms of input data, GPUs, energy, and training. The classical approach was to understand how genes transcribe to mRNA, and how mRNA translates to polypeptides; how those are cleaved by the cell, and fold in 3D space; and how those 3D shapes results in actual biological function. It required real-world measurement, experiment, and modeling in silico using biophysical models. Those are all hard research efforts. And it seems like the mindset now is: we've done enough hard research, let's feed what we know into a model, hope we've chosen the right hyperparameters, and see what we get. Hidden in the weights and biases of the model will be that deeper map of the real world that we have not yet fully grasped through research. But the AI cannot provide a 'why'. Its network of weights and biases are as unintelligible to us as the underlying scientific principles of the real world we gave up trying to understand along the way. When AI produces a result that is surprising, we still have to validate it in the real world, and work backwards through the hard research to understand why we are surprised. If AI is just a tool for a shotgun approach to discovery, that may be fine. However, I fear it is sucking a lot of air out of the room from the classical approaches. When 'AI' produces incorrect, misleading, or underwhelming results? Well, throw more GPUs at it; more tokens; more joules; more parameters. We have blind faith it'll work itself out. But because the AI can never provide a guarantee of correctness, it is only useful to those with the infrastructure to carry out those real-world validations on its output, so it's not really going to create a paradigm shift. It can provide only a marginal improvement at the top of the funnel for existing discovery pipelines. And because AI is very expensive and getting more so, there's a pretty hard cap on how valuable it would be to a drugmaker. I know I'm not the only one worried about a bubble here.
- wizzwizz4 2y ago
  Hopefully the bubble will pop when the GenAI bubble does. (Not that it probably should, since this both predates and is unrelated to it… but hype isn't rational to begin with.)
- sweezyjeezy 2y ago
  You're using "AI" quite broadly here. Here's a perspective from computer vision (my field). For decades, CV was focused on trying to 'understand' how to do the task. This meant a lot of hand crafting of low level features that are common in images, finding clever ways to make them invariant to typical 3D transformations. This works well for some tasks, and is still used today in things like robotics, SLAM etc. However - when we then want to add an extra level of complexity - e.g. to try and model an abstract concept like "cat", we hit a bit of a brick wall. This happens to be a task where feeding a large dataset into an (mostly) unconstrained machine learning model does very well. > The classical approach was to understand how genes transcribe to mRNA, and how mRNA translates to polypeptides; how those are cleaved by the cell, and fold in 3D space; and how those 3D shapes results in actual biological function. I don't have the expertise to critique this, but it does sound like we're in the extreme 'high complexity' zone to me. Some questions for you: - how accurate does each stage of this need to get to useful performance? Are you sure there are no brick walls here? How long do you think this approach will take to deliver results? - do you not have to validate a surprising classical finding in the same way that you would an AI model - i.e. how much does the "why" matter? "the AI can never provide a guarantee of correctness" - is true, but what it was merely extremely accurate, in the same way that many computer vision models are?
nightowl_games 2y ago
Pretty sure the All In pod guys were using this company as an example of the AI bubble being over inflated. "It's just three guys and a model and they think it's worth X hundred million"
yldedly 2y ago
"Rives sees ESM3’s generation of new proteins by iterating through various sequences as analogous to evolution." Except for the part where a sequence is actually deemed more fit, ie natural selection? And the part where mutations are random, instead of sampled from the training data manifold, so much more constrained? ...so really it's a worse version of random search?
yieldcrv 2y ago
distribute protein folding was a waste of time and energy we just had to wait for this approach to be apparent

124 points by gmays 2y ago

Tangential - the laws of nature discovered by our brain usually involve just few quantities, like f=ma. On one side it is a great ability of our brain for analytical reduction, on the other side it is just an inability to deal with complex multiparameter phenomena without such a reduction. I wonder if pumping more and more data into the NNs we'd be able to distill emerging multiparameter correlations which happen to be new laws of nature irreducible to more simpler ones.

spywaregorilla 2y ago
Eh, not in this context at least. Nature is a big term. Physics just "is". This makes it a good place for simple mechanics to appear and be derived. You can argue around how universal that is all you want but proteins themselves are the result of an enormously large search of a hugely high dimensional space. It doesn't seem possible for them to ever be reduced.
ben_w 2y ago
I disagree, our brains almost always deal with complex multiparameter phenomena. The part of us that can't cope with that is our System 2 reasoning: our minds' rules based thinking is great at F=ma but cannot enumerate the rules necessary at the level of retina cell activation patterns to recognise a tiger; our System 1 can, and always does, we just call it "intuition" or "common sense" or "a gut feeling". https://en.wikipedia.org/wiki/Dual_process_theory https://en.wikipedia.org/wiki/Dual_process_theory
akreal 2y ago
It took the modern human brain quite a bit of observations to discover and formulate f=ma, it was originally published in 1687.

btbuildem 2y ago

I wonder if this will ever come full circle (or.. spiral) and the AI tools we've created will in turn lead the way to discovering / inventing new proteins / cells / life forms that eventually outsmart and outcompete us.

szundi 2y ago
Isn’t the Terminator good enough?
- warkdarrior 2y ago
  As usual, life is stranger than fiction.

yeutterg 2y ago

Just want to congratulate Tom Hayes on the big launch!

Spacecosmonaut 2y ago

Very nice work. We need brighter fluorescent protein tags that are more compact, in particular in the far red spectrum. The size of current fluorescent protein coding DNA sequences is out of reach of prime editing and still relies on less efficient gene editing technology.

alan-hn 2y ago
But they aren't brighter...
- changoplatanero 2y ago
  its not more compact either, right?
- Spacecosmonaut 2y ago
  There is only so much that you can get out of tinkering with naturally evolved proteins. I suspect that these kind of tools can be used to generate smaller & brighter FP's in the near future.

thebeardisred 2y ago

> For a smaller open-source version, certain sequences, such as those from viruses and a US government list of worrying pathogens and toxins, were excluded from training. Neither can ESM3-open — which scientists anywhere can download and run independently — be prompted to generate such proteins. That sounds like a glove being thrown down.

trott 2y ago

> matching less than 60% of the sequence of the most closely related fluorescent protein > When the researchers made around 100 of the resulting designs, several were as bright as natural GFPs, which are still vastly dimmer than lab-engineered variants. So they didn't come up with better functionality, unlike what some commentators imply. They basically introduced a bunch of mutations while preserving the overall function. Relevant: https://en.wikipedia.org/wiki/Conservative_replacement https://en.wikipedia.org/wiki/Conservative_replacement

civilized 2y ago
If the mutations were non-synonymous, resulting in different amino acids, the fact that they keep the natural function is still kinda cool. Very much a pure research result AFAICT, but worth a little something.
- flobosg 2y ago
  This is already a well known fact: protein structure (and consequently function) is much more conserved than sequence, mostly due to biophysical constraints.
  - civilized 2y ago
    Forgive my ignorance - is that true even for non-synonymous mutations? They usually have a hard time disrupting the big structure?
    - flobosg 2y ago
      If non-synonymous mutations do not change the biophysical features of the amino acid residues, then the structure is usually kept. Alternatively, it can be the case that a disruptive mutation is compensated by another one that keeps the structure/function/phenotype. This is the basis for evolutionary coupling based structure prediction methods, such as Alphafold.
- dekhn 2y ago

flobosg 2y ago

> However, its amino-acid sequence is vastly different, matching less than 60% of the sequence of the most closely related fluorescent protein in its training data set. Not to downplay this achievement, but 60% sequence identity is nowhere near “vastly different”.

trott 2y ago
The twilight zone is 20-35%: https://pubmed.ncbi.nlm.nih.gov/10195279/ https://pubmed.ncbi.nlm.nih.gov/10195279/ (Incidentally, the author was on my thesis committee, but this isn't precisely my field of expertise.)
- flobosg 2y ago
  Proteins from the same family (and thus the same fold) can share less than 10% identity and still keep the same functionality as shown by e.g. profile Hidden Markov Model comparisons.

throwaway24124 2y ago

Are there any good resources for understanding models like this? Specifically a "protein language model". I have a basic grasp on how LLMs tokenize and encode natural language, but what does a protein language actually look like? An LLM can produce results that look correct but are actually incorrect, how are proteins produced by this model validated? Are the outputs run through some other software to determine whether the proteins are valid?

cottonseed 2y ago
Proteins are linear molecules consisting of sequences of (mostly) 20 amino acids. You can see the list of amino acids here: https://en.wikipedia.org/wiki/Amino_acid#Table_of_standard_amino_acid_abbreviations_and_properties https://en.wikipedia.org/wiki/Amino_acid#Table_of_standard_a.... There is a standard encoding of amino acids using single letters, A for alanine, etc. Earlier versions of ESM (I haven't read the ESM3 paper yet) uses one token per amino acid, plus a few control tokens (beginning of sequence, end of sequence, class token, mask, etc.) Earlier versions of ESM were BERT-style models focused on understanding, not GPT-style generative models.
hn_throwaway_99 2y ago
Agreed, would be interested if someone with more knowledge could comment. My layman's understanding of LLMs is that they are essentially "fancy autocomplete". That is, you take a whole corpus of text, then train the model to determine the statistical relationships between those words (more accurately, tokens), so that given a list of tokens of length N, the LLM will find the next most likely token for N + 1, and then to generate whole sentences/paragraphs, you just recursively repeat this process. I certainly understand encoding proteins as just a linear sequence of tokens representing their amino acids, but how does that then map to a human-language description of the function of those proteins?
- changoplatanero 2y ago
  Most protein language models are not able to understand human-language descriptions of proteins. Mostly they just predict the next amino acid in a sequence and sometimes they can understand certain structured metadata tags.

brcmthrowaway 2y ago

Can protein models help cure prion diseases?

Exoristos 2y ago
At present this is unclear. But on the other hand, think of all the potential new prion diseases!

zacharyvoase 2y ago

Reading this it sounds like 'AI' is when you build a heuristic model (which we've had for a while now) but pass some threshold of cost in terms of input data, GPUs, energy, and training. The classical approach was to understand how genes transcribe to mRNA, and how mRNA translates to polypeptides; how those are cleaved by the cell, and fold in 3D space; and how those 3D shapes results in actual biological function. It required real-world measurement, experiment, and modeling in silico using biophysical models. Those are all hard research efforts. And it seems like the mindset now is: we've done enough hard research, let's feed what we know into a model, hope we've chosen the right hyperparameters, and see what we get. Hidden in the weights and biases of the model will be that deeper map of the real world that we have not yet fully grasped through research. But the AI cannot provide a 'why'. Its network of weights and biases are as unintelligible to us as the underlying scientific principles of the real world we gave up trying to understand along the way. When AI produces a result that is surprising, we still have to validate it in the real world, and work backwards through the hard research to understand why we are surprised. If AI is just a tool for a shotgun approach to discovery, that may be fine. However, I fear it is sucking a lot of air out of the room from the classical approaches. When 'AI' produces incorrect, misleading, or underwhelming results? Well, throw more GPUs at it; more tokens; more joules; more parameters. We have blind faith it'll work itself out. But because the AI can never provide a guarantee of correctness, it is only useful to those with the infrastructure to carry out those real-world validations on its output, so it's not really going to create a paradigm shift. It can provide only a marginal improvement at the top of the funnel for existing discovery pipelines. And because AI is very expensive and getting more so, there's a pretty hard cap on how valuable it would be to a drugmaker. I know I'm not the only one worried about a bubble here.

wizzwizz4 2y ago
Hopefully the bubble will pop when the GenAI bubble does. (Not that it probably should, since this both predates and is unrelated to it… but hype isn't rational to begin with.)
sweezyjeezy 2y ago
You're using "AI" quite broadly here. Here's a perspective from computer vision (my field). For decades, CV was focused on trying to 'understand' how to do the task. This meant a lot of hand crafting of low level features that are common in images, finding clever ways to make them invariant to typical 3D transformations. This works well for some tasks, and is still used today in things like robotics, SLAM etc. However - when we then want to add an extra level of complexity - e.g. to try and model an abstract concept like "cat", we hit a bit of a brick wall. This happens to be a task where feeding a large dataset into an (mostly) unconstrained machine learning model does very well. > The classical approach was to understand how genes transcribe to mRNA, and how mRNA translates to polypeptides; how those are cleaved by the cell, and fold in 3D space; and how those 3D shapes results in actual biological function. I don't have the expertise to critique this, but it does sound like we're in the extreme 'high complexity' zone to me. Some questions for you: - how accurate does each stage of this need to get to useful performance? Are you sure there are no brick walls here? How long do you think this approach will take to deliver results? - do you not have to validate a surprising classical finding in the same way that you would an AI model - i.e. how much does the "why" matter? "the AI can never provide a guarantee of correctness" - is true, but what it was merely extremely accurate, in the same way that many computer vision models are?

nightowl_games 2y ago

Pretty sure the All In pod guys were using this company as an example of the AI bubble being over inflated. "It's just three guys and a model and they think it's worth X hundred million"

yldedly 2y ago

"Rives sees ESM3’s generation of new proteins by iterating through various sequences as analogous to evolution." Except for the part where a sequence is actually deemed more fit, ie natural selection? And the part where mutations are random, instead of sampled from the training data manifold, so much more constrained? ...so really it's a worse version of random search?

yieldcrv 2y ago

distribute protein folding was a waste of time and energy we just had to wait for this approach to be apparent