4 ms·

Some notable technical information: It is a 1.2bln param model, trained on raw sequences only, can generate full length functional proteins of about 150-200 res

Some notable technical information: It is a 1.2bln param model, trained on raw sequences only, can generate full length functional proteins of about 150-200 residues (approx lysozyme size). The generated proteins are very different to native ones (30-40% similarity). The interesting thing about this model is that it also exhibit emergent capabilities. It was trained only on raw sequences but somehow managed to capture information about functionality and solubility of the folded proteins, and then implemented that in the generated sequences. Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". This is probably the model managed to learn protein grammar purely by brute force. Now if only we can get a model in the range of 100bln parameters...

dr_kiszonka 3y ago
> Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". Thanks for a very informative comment! Would you have a minute to give a concrete example of such a sentence (something that could be in a training set)? Whenever I add "folding" to queries containing a variation of amino acids, Google starts showing images instead of sequences. (I have no background in bioinformatics.)
_hl_ 3y ago
> Now if only we can get a model in the range of 100bln parameters Do we have enough data to train such a large model in a meaningful way?
- a_bonobo 3y ago
  we've gotten pretty good at predicting proteins from genome assemblies, but we don't have the manpower to manually curate that data. Swissprot, the project that manually curates and scores protein sequences, has about half a million 'supported' protein models in its last release: https://web.expasy.org/docs/relnotes/relstat.html https://web.expasy.org/docs/relnotes/relstat.html TrEMBL, the database that has curated + uncurated, has 249,308,459 proteins in its last release: https://www.ebi.ac.uk/uniprot/TrEMBLstats https://www.ebi.ac.uk/uniprot/TrEMBLstats Generating uncurated protein sequences has become fairly easy - I can generate a draft genome assembly in about a day, predicting proteins using MAKER or BRAKEr or other HMM-based pipelines takes another day or so. Most eukaryotes have about 20,000 to 30,000 genes that translate into proteins. I'm involved in a project that assembled 300 draft genomes in the last 6 months- now we have between 6 to 9 million proteins (most of them not 'novel' in function - novel genes/proteins are rare, and a HMM trained on known sequences cannot easily find novel genes).
  - a_bonobo 3y ago
    Addendum: To train an LLM you want 'diverse' training data, right? Not just Reddit posts but also technical manuals. In protein space I'm confident we've sequenced the majority of what is out there. For the majority we don't know what they do, but we know that they are there (being expressed in the host). In every genome sequencing project we can find a similar looking gene for >95% of genes, with much of the remaining 5% being noise or contaminants. To train LLMs we can't really generate more 'new' data in this space; this is it! Even 'my' 6 to 9 million projects won't add much new information, at most the LLMs will know the 'space' of possible changes for those existing proteins.
    - tomohelix 3y ago
      I think English is also just the rehash of the same 2000-3000 words. The ability to meaningfully arrange those words in new ways and in longer sentences are what make the current LLMs so powerful. In that way, these protein models can still learn more from the new data but they will definitely need some sort of reinforcement learning and grounding. That is where wetlab scientists will have to do. Verifying the generated protein is functional and stable, i.e. "making sense", is where we will need more data. Only a miniscule amount of protein has verified structure and function. While an HMM can predict their functions pretty well based on homology, it can often be wrong. Having that kind of noise in the training data can degrade the performance of these protein models. Perhaps we need better curated dataset before we can scale these up.
    - inciampati 3y ago
      We have only scratched the surface of what's out there. Probably it's unlikely that we will find totally novel proteins. But it's a mistake to forget about prokaryotes and phages. Everywhere we look there is something new. The oceans are vast and full of life. And we have barely begun to dig into the life deeper in the crust. Seeing how all of this varies will only make models stronger. It's an enormous collection relative to human language.
noduerme 3y ago
> purely by brute force the new underlying model of the world. First it was: Who needs to understand an algorithm when you can just simulate everything? Now it's: Who needs to simulate everything when you can just let a large enough black box approximate a solution?
- teekert 3y ago
  It’s only a black box because our brains aren’t good enough.
  - detrites 3y ago
    But if our brains came up with the black box, can that really be true?
    - daveguy 3y ago
      Yes. All that is required to make that true is that our brains come up with a model that we cannot comprehend or that reflects a reality that we do not understand. That has been true since cycles and epicycles were used to describe the motion of celestial bodies and probably long before that.
- throwawaymaths 3y ago
  > Who needs to simulate everything when you can just let a large enough black box approximate a solution You are kind of describing science here. Take chemistry, it's a series of black boxes: chemical reaction patterns, bohr model, vsepr, orbital shell theory, perturbation theory

dr_kiszonka 3y ago

> Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". Thanks for a very informative comment! Would you have a minute to give a concrete example of such a sentence (something that could be in a training set)? Whenever I add "folding" to queries containing a variation of amino acids, Google starts showing images instead of sequences. (I have no background in bioinformatics.)

_hl_ 3y ago

> Now if only we can get a model in the range of 100bln parameters Do we have enough data to train such a large model in a meaningful way?

a_bonobo 3y ago
we've gotten pretty good at predicting proteins from genome assemblies, but we don't have the manpower to manually curate that data. Swissprot, the project that manually curates and scores protein sequences, has about half a million 'supported' protein models in its last release: https://web.expasy.org/docs/relnotes/relstat.html https://web.expasy.org/docs/relnotes/relstat.html TrEMBL, the database that has curated + uncurated, has 249,308,459 proteins in its last release: https://www.ebi.ac.uk/uniprot/TrEMBLstats https://www.ebi.ac.uk/uniprot/TrEMBLstats Generating uncurated protein sequences has become fairly easy - I can generate a draft genome assembly in about a day, predicting proteins using MAKER or BRAKEr or other HMM-based pipelines takes another day or so. Most eukaryotes have about 20,000 to 30,000 genes that translate into proteins. I'm involved in a project that assembled 300 draft genomes in the last 6 months- now we have between 6 to 9 million proteins (most of them not 'novel' in function - novel genes/proteins are rare, and a HMM trained on known sequences cannot easily find novel genes).
- a_bonobo 3y ago
  Addendum: To train an LLM you want 'diverse' training data, right? Not just Reddit posts but also technical manuals. In protein space I'm confident we've sequenced the majority of what is out there. For the majority we don't know what they do, but we know that they are there (being expressed in the host). In every genome sequencing project we can find a similar looking gene for >95% of genes, with much of the remaining 5% being noise or contaminants. To train LLMs we can't really generate more 'new' data in this space; this is it! Even 'my' 6 to 9 million projects won't add much new information, at most the LLMs will know the 'space' of possible changes for those existing proteins.
  - tomohelix 3y ago
    I think English is also just the rehash of the same 2000-3000 words. The ability to meaningfully arrange those words in new ways and in longer sentences are what make the current LLMs so powerful. In that way, these protein models can still learn more from the new data but they will definitely need some sort of reinforcement learning and grounding. That is where wetlab scientists will have to do. Verifying the generated protein is functional and stable, i.e. "making sense", is where we will need more data. Only a miniscule amount of protein has verified structure and function. While an HMM can predict their functions pretty well based on homology, it can often be wrong. Having that kind of noise in the training data can degrade the performance of these protein models. Perhaps we need better curated dataset before we can scale these up.
  - inciampati 3y ago
    We have only scratched the surface of what's out there. Probably it's unlikely that we will find totally novel proteins. But it's a mistake to forget about prokaryotes and phages. Everywhere we look there is something new. The oceans are vast and full of life. And we have barely begun to dig into the life deeper in the crust. Seeing how all of this varies will only make models stronger. It's an enormous collection relative to human language.

noduerme 3y ago

> purely by brute force the new underlying model of the world. First it was: Who needs to understand an algorithm when you can just simulate everything? Now it's: Who needs to simulate everything when you can just let a large enough black box approximate a solution?

teekert 3y ago
It’s only a black box because our brains aren’t good enough.
- detrites 3y ago
  But if our brains came up with the black box, can that really be true?
  - daveguy 3y ago
    Yes. All that is required to make that true is that our brains come up with a model that we cannot comprehend or that reflects a reality that we do not understand. That has been true since cycles and epicycles were used to describe the motion of celestial bodies and probably long before that.
throwawaymaths 3y ago
> Who needs to simulate everything when you can just let a large enough black box approximate a solution You are kind of describing science here. Take chemistry, it's a series of black boxes: chemical reaction patterns, bohr model, vsepr, orbital shell theory, perturbation theory