5 ms·

Large language models generate functional protein sequences across families

172 points by samwillis 3y ago

deleted 3y ago
[deleted]
samwillis 3y ago
PDF: http://cdn.fraserlab.com/publications/2023_madani.pdf http://cdn.fraserlab.com/publications/2023_madani.pdf Code: https://github.com/salesforce/progen https://github.com/salesforce/progen
- arthur2e5 3y ago
  Pretty sure the preprint is at https://www.biorxiv.org/content/10.1101/2021.07.18.452833v1.full https://www.biorxiv.org/content/10.1101/2021.07.18.452833v1..... Good for checking out what the editors have done.
hecanjog 3y ago
I'm surprised that Salesforce has a research division, and they're working on something like this.
- sb8244 3y ago
  I know that Salesforce is committed to the 1% initiative, so maybe this falls into that. 1% can do a lot at their revenue.
- ipnon 3y ago
  The game theory of useless corporate research departments is that by spinning plates for you they’re not building moats for your competitors. There is quite a lot of money to be saved by essentially nerd sniping with large wads of cash.
sroussey 3y ago
2022, btw
- tomohelix 3y ago
  >Published: 26 January 2023 Well, technically it is out in 2023 but sure, you can argue it was probably in the prepub state since 2022...
  - sroussey 3y ago
    I was supposed to be reply to another comment. The GitHub is from 2022: https://github.com/salesforce/progen https://github.com/salesforce/progen
tomohelix 3y ago
Some notable technical information: It is a 1.2bln param model, trained on raw sequences only, can generate full length functional proteins of about 150-200 residues (approx lysozyme size). The generated proteins are very different to native ones (30-40% similarity). The interesting thing about this model is that it also exhibit emergent capabilities. It was trained only on raw sequences but somehow managed to capture information about functionality and solubility of the folded proteins, and then implemented that in the generated sequences. Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". This is probably the model managed to learn protein grammar purely by brute force. Now if only we can get a model in the range of 100bln parameters...
- dr_kiszonka 3y ago
  > Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". Thanks for a very informative comment! Would you have a minute to give a concrete example of such a sentence (something that could be in a training set)? Whenever I add "folding" to queries containing a variation of amino acids, Google starts showing images instead of sequences. (I have no background in bioinformatics.)
- _hl_ 3y ago
  > Now if only we can get a model in the range of 100bln parameters Do we have enough data to train such a large model in a meaningful way?
  - a_bonobo 3y ago
    we've gotten pretty good at predicting proteins from genome assemblies, but we don't have the manpower to manually curate that data. Swissprot, the project that manually curates and scores protein sequences, has about half a million 'supported' protein models in its last release: https://web.expasy.org/docs/relnotes/relstat.html https://web.expasy.org/docs/relnotes/relstat.html TrEMBL, the database that has curated + uncurated, has 249,308,459 proteins in its last release: https://www.ebi.ac.uk/uniprot/TrEMBLstats https://www.ebi.ac.uk/uniprot/TrEMBLstats Generating uncurated protein sequences has become fairly easy - I can generate a draft genome assembly in about a day, predicting proteins using MAKER or BRAKEr or other HMM-based pipelines takes another day or so. Most eukaryotes have about 20,000 to 30,000 genes that translate into proteins. I'm involved in a project that assembled 300 draft genomes in the last 6 months- now we have between 6 to 9 million proteins (most of them not 'novel' in function - novel genes/proteins are rare, and a HMM trained on known sequences cannot easily find novel genes).
lysozyme 3y ago
When evaluating this work, it’s important to remember that the functional labels and protein family assignments on each of the 280 million input sequences were originally assigned by an HMM model using human curated sequence groups as part of the pfam project, so the model is predicting a prediction (or perhaps conditioned on a prediction would be more accurate). Furthermore, the authors must engage a lot of human curation to ensure the sequences they generate are active. First, they pick an easy target. Second, they employ by-hand classical bioinformatics techniques on their predicted sequences after they are generated. For example, they manually align them and select those which contain specific important amino acids at specific positions which are present in 100% of functional proteins of that class, and are required for function. This is all done by a human bioinformatics expert (or automated) before they test the generated sequences. This is the protein equivalent of cherry-picking great examples of, for example, ChatGPT responses and presenting them as if the model only made predictions like that. One other comment, in protein science, a sequence with 40% identity to another sequence is not “very different” if it is homologous. Since this model is essentially generating homologs from a particular class, it’s no surprise at a pairwise amino acid level, the generated sequences have this degree of similarity. Take proteins in any functional family and compare them. They will have the same overall 3-D structure—called their “fold”—yet have pairwise sequence identities much lower than 30–40%. This “degeneracy”, the notion that there are many diverse sequences that all fold into the same shape, is both a fundamental empirical observation in protein science as well as a grounded physical theory. Not to be negative. I really enjoyed reading this paper and I think the work is important. Some related work by Meta AI is the ESM series of models [1] trained on the same data (the UniProt dataset [2]). One thing I wonder is about the vocabulary size of this model. The number of tokens is 26 for the 20 amino acids and some extras, whereas for a LLM like Meta’s LLaMa the vocab size is 32,000. I wonder how that changes training and inference, and how we can adopt the transformer architecture for this scenario. 1. https://github.com/facebookresearch/esm https://github.com/facebookresearch/esm 2. https://www.uniprot.org/help/downloads https://www.uniprot.org/help/downloads
- tomohelix 3y ago
  I consider all the manual curation effectively a form of RLHF that can be imposed automatically later on. We saw how much this can improve a raw LLM by looking at the output of ChatGPT. Otherwise, the criticism of LLMs being just glorified autocomplete machines isn't that far from reality. In other words, it is just an expected requirement for LLMs to be effective. You are probably right that lysozyme is an easy target and may have large sequence variety between homologs so saying "very different" for 30-40% is not correct. But that is only in the context of biology and protein structure and function. This is an LLM trained on primary sequences only. It doesn't know anything about the folds or domains or functional sites (unless I am wrong and those are part of the metadata fed to it during training). Yet it did learn enough to generalize to the point that even with only 30-40% identity, it still produces soluble proteins with the same function. I am sure you know that at 40% differences, one protein can be in an entirely different superfamily from another. So it is still an impressively low identity score. Also, I think it is more appropriate to compare the amino acids to things like the alphabets than vocabs. Domains would probably be an equivalent to LLaMa vocab.
ajuc 3y ago
AI generating a virus/prion/whatever that we synthesize without understanding what it does is the easiest way to the bad singularity people were warning us about.
- cyanydeez 3y ago
  The bad singularity is for those who can't afford the coming life subscription plan.
  - PartiallyTyped 3y ago
    Context: https://youtu.be/IFe9wiDfb0E https://youtu.be/IFe9wiDfb0E
cyanydeez 3y ago
Ooh, can't wait for generative cancer. Thanks AI.
a_bonobo 3y ago
There's a LOT of interesting stuff happening with large language models in biology: Transformer trained on human genomes, learns to identify different genomic elements like enhancers without knowing what enhancers are https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2 https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2 Similarly, a transformer trained on only plant genomes 'knows' how the strength of genomic variants' impact https://www.biorxiv.org/content/10.1101/2022.08.22.504706v2 https://www.biorxiv.org/content/10.1101/2022.08.22.504706v2 Lots of experiments happening around GPT-4 too; Using regular GPT-4 to curate cell type annotations https://www.biorxiv.org/content/10.1101/2023.04.16.537094v1 https://www.biorxiv.org/content/10.1101/2023.04.16.537094v1 Testing GPT-4 across a variety of 'standard' tasks like linking gene IDs to protein IDs https://www.biorxiv.org/content/10.1101/2023.03.11.532238v1 https://www.biorxiv.org/content/10.1101/2023.03.11.532238v1
- seydor 3y ago
  It would be super interesting if we figure out that everything in life including language are fractals of molecular biology
  - deleted 3y ago
    [deleted]
intalentive 3y ago
That “large language models” are unfortunately named becomes clearer when they’re applied to non-language domains. They’re auto-associative models that learn mappings between discrete elements in a sequence or set.
CatsAndCoding 3y ago
This is so coool!

172 points by samwillis 3y ago

deleted 3y ago

[deleted]

PDF: http://cdn.fraserlab.com/publications/2023_madani.pdf http://cdn.fraserlab.com/publications/2023_madani.pdf Code: https://github.com/salesforce/progen https://github.com/salesforce/progen

arthur2e5 3y ago
Pretty sure the preprint is at https://www.biorxiv.org/content/10.1101/2021.07.18.452833v1.full https://www.biorxiv.org/content/10.1101/2021.07.18.452833v1..... Good for checking out what the editors have done.

hecanjog 3y ago

I'm surprised that Salesforce has a research division, and they're working on something like this.

sb8244 3y ago
I know that Salesforce is committed to the 1% initiative, so maybe this falls into that. 1% can do a lot at their revenue.
ipnon 3y ago
The game theory of useless corporate research departments is that by spinning plates for you they’re not building moats for your competitors. There is quite a lot of money to be saved by essentially nerd sniping with large wads of cash.

sroussey 3y ago

2022, btw

tomohelix 3y ago
>Published: 26 January 2023 Well, technically it is out in 2023 but sure, you can argue it was probably in the prepub state since 2022...
- sroussey 3y ago
  I was supposed to be reply to another comment. The GitHub is from 2022: https://github.com/salesforce/progen https://github.com/salesforce/progen

tomohelix 3y ago

Some notable technical information: It is a 1.2bln param model, trained on raw sequences only, can generate full length functional proteins of about 150-200 residues (approx lysozyme size). The generated proteins are very different to native ones (30-40% similarity). The interesting thing about this model is that it also exhibit emergent capabilities. It was trained only on raw sequences but somehow managed to capture information about functionality and solubility of the folded proteins, and then implemented that in the generated sequences. Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". This is probably the model managed to learn protein grammar purely by brute force. Now if only we can get a model in the range of 100bln parameters...

dr_kiszonka 3y ago
> Amino acid sequences are just a bunch of jumbled words if you compared them to English. It usually has to go through folding to form proper "sentences" with meanings. I guess you can compare this to "grammar". Thanks for a very informative comment! Would you have a minute to give a concrete example of such a sentence (something that could be in a training set)? Whenever I add "folding" to queries containing a variation of amino acids, Google starts showing images instead of sequences. (I have no background in bioinformatics.)
_hl_ 3y ago
> Now if only we can get a model in the range of 100bln parameters Do we have enough data to train such a large model in a meaningful way?
- a_bonobo 3y ago
  we've gotten pretty good at predicting proteins from genome assemblies, but we don't have the manpower to manually curate that data. Swissprot, the project that manually curates and scores protein sequences, has about half a million 'supported' protein models in its last release: https://web.expasy.org/docs/relnotes/relstat.html https://web.expasy.org/docs/relnotes/relstat.html TrEMBL, the database that has curated + uncurated, has 249,308,459 proteins in its last release: https://www.ebi.ac.uk/uniprot/TrEMBLstats https://www.ebi.ac.uk/uniprot/TrEMBLstats Generating uncurated protein sequences has become fairly easy - I can generate a draft genome assembly in about a day, predicting proteins using MAKER or BRAKEr or other HMM-based pipelines takes another day or so. Most eukaryotes have about 20,000 to 30,000 genes that translate into proteins. I'm involved in a project that assembled 300 draft genomes in the last 6 months- now we have between 6 to 9 million proteins (most of them not 'novel' in function - novel genes/proteins are rare, and a HMM trained on known sequences cannot easily find novel genes).

lysozyme 3y ago

When evaluating this work, it’s important to remember that the functional labels and protein family assignments on each of the 280 million input sequences were originally assigned by an HMM model using human curated sequence groups as part of the pfam project, so the model is predicting a prediction (or perhaps conditioned on a prediction would be more accurate). Furthermore, the authors must engage a lot of human curation to ensure the sequences they generate are active. First, they pick an easy target. Second, they employ by-hand classical bioinformatics techniques on their predicted sequences after they are generated. For example, they manually align them and select those which contain specific important amino acids at specific positions which are present in 100% of functional proteins of that class, and are required for function. This is all done by a human bioinformatics expert (or automated) before they test the generated sequences. This is the protein equivalent of cherry-picking great examples of, for example, ChatGPT responses and presenting them as if the model only made predictions like that. One other comment, in protein science, a sequence with 40% identity to another sequence is not “very different” if it is homologous. Since this model is essentially generating homologs from a particular class, it’s no surprise at a pairwise amino acid level, the generated sequences have this degree of similarity. Take proteins in any functional family and compare them. They will have the same overall 3-D structure—called their “fold”—yet have pairwise sequence identities much lower than 30–40%. This “degeneracy”, the notion that there are many diverse sequences that all fold into the same shape, is both a fundamental empirical observation in protein science as well as a grounded physical theory. Not to be negative. I really enjoyed reading this paper and I think the work is important. Some related work by Meta AI is the ESM series of models [1] trained on the same data (the UniProt dataset [2]). One thing I wonder is about the vocabulary size of this model. The number of tokens is 26 for the 20 amino acids and some extras, whereas for a LLM like Meta’s LLaMa the vocab size is 32,000. I wonder how that changes training and inference, and how we can adopt the transformer architecture for this scenario. 1. https://github.com/facebookresearch/esm https://github.com/facebookresearch/esm 2. https://www.uniprot.org/help/downloads https://www.uniprot.org/help/downloads

tomohelix 3y ago
I consider all the manual curation effectively a form of RLHF that can be imposed automatically later on. We saw how much this can improve a raw LLM by looking at the output of ChatGPT. Otherwise, the criticism of LLMs being just glorified autocomplete machines isn't that far from reality. In other words, it is just an expected requirement for LLMs to be effective. You are probably right that lysozyme is an easy target and may have large sequence variety between homologs so saying "very different" for 30-40% is not correct. But that is only in the context of biology and protein structure and function. This is an LLM trained on primary sequences only. It doesn't know anything about the folds or domains or functional sites (unless I am wrong and those are part of the metadata fed to it during training). Yet it did learn enough to generalize to the point that even with only 30-40% identity, it still produces soluble proteins with the same function. I am sure you know that at 40% differences, one protein can be in an entirely different superfamily from another. So it is still an impressively low identity score. Also, I think it is more appropriate to compare the amino acids to things like the alphabets than vocabs. Domains would probably be an equivalent to LLaMa vocab.

ajuc 3y ago

AI generating a virus/prion/whatever that we synthesize without understanding what it does is the easiest way to the bad singularity people were warning us about.

cyanydeez 3y ago
The bad singularity is for those who can't afford the coming life subscription plan.
- PartiallyTyped 3y ago
  Context: https://youtu.be/IFe9wiDfb0E https://youtu.be/IFe9wiDfb0E

cyanydeez 3y ago

Ooh, can't wait for generative cancer. Thanks AI.

a_bonobo 3y ago

There's a LOT of interesting stuff happening with large language models in biology: Transformer trained on human genomes, learns to identify different genomic elements like enhancers without knowing what enhancers are https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2 https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2 Similarly, a transformer trained on only plant genomes 'knows' how the strength of genomic variants' impact https://www.biorxiv.org/content/10.1101/2022.08.22.504706v2 https://www.biorxiv.org/content/10.1101/2022.08.22.504706v2 Lots of experiments happening around GPT-4 too; Using regular GPT-4 to curate cell type annotations https://www.biorxiv.org/content/10.1101/2023.04.16.537094v1 https://www.biorxiv.org/content/10.1101/2023.04.16.537094v1 Testing GPT-4 across a variety of 'standard' tasks like linking gene IDs to protein IDs https://www.biorxiv.org/content/10.1101/2023.03.11.532238v1 https://www.biorxiv.org/content/10.1101/2023.03.11.532238v1

seydor 3y ago
It would be super interesting if we figure out that everything in life including language are fractals of molecular biology
- deleted 3y ago
  [deleted]

intalentive 3y ago

That “large language models” are unfortunately named becomes clearer when they’re applied to non-language domains. They’re auto-associative models that learn mappings between discrete elements in a sequence or set.

CatsAndCoding 3y ago

This is so coool!