6 ms·

I continue to be impressed by how quickly DeepMind has managed to progress in such a short time. CASP13 was a shocker to all of us I think, but many were skepti

I continue to be impressed by how quickly DeepMind has managed to progress in such a short time. CASP13 was a shocker to all of us I think, but many were skeptical as to the longevity of the performance DeepMind was able to achieve. I believe with CASP14 rankings now released, it's safe to say that they've proven themselves. Congratulations to the team! This work will have far reaching impacts, and I hope that you continue to invest heavily in this area of research.

whimsicalism 6y ago
Progress like this was, in my view, inevitable after the invention of unsupervised transformers. It'll be genetics next. e: although AlphaFold appears to be convolutionally based! I suspect that'll change soon.
- klmr 6y ago
  > It'll be genetics next. Which part of genetics are you thinking of? Much of genetics isn’t amenable to this kind of ML, because it isn’t some kind of optimisation problem. And many other parts don’t require ML because they can be modelled very closely using exact methods. ML does get used here, and sometimes to great effect (e.g. DeepVariant, which often outperforms other methods, but not by much — not because DeepVariant isn’t good, but rather because we have very efficient approximations to the exact solution).
  - whimsicalism 6y ago
    What do you mean? Genetics is amenable because the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network. There are plenty of inferences that you would want to do on genetic sequences that we can't model exactly and there is some past work on doing stuff like this, although biology is usually a few years behind. https://www.nature.com/articles/s41592-018-0138-4 https://www.nature.com/articles/s41592-018-0138-4 e: for clarity
    - garmaine 6y ago
      This is word salad.
      - whimsicalism 6y ago
        Rude. I would appreciate substantive criticism, especially when I'm linking papers in Nature starting to do exactly what I'm talking about.
        garmaine 6y ago
        I cannot give constructive feedback to something which is incomprehensible. "the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network" The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important. Language modeling in the context of machine learning typically means NLP methods. Genetics is nothing like natural language. Auto-regression is using (typically time series) information to predict the next codon. This makes very little sense in the context of genetics since, again, the genetic code is not an information carrying medium in the same sense as human language. Being able to predict the next codon tells you zilch in terms of useable information. "Depth of understanding by the network" ... what does that even mean??? The above sentence is a bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way. AKA word salad.
        whimsicalism 6y ago
        > The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important. aka a sequence. "a book is not a sequence so much as a discrete set of chapters which are themselves sequences of paragraphs which are themselves sequences of sentences" -> still a sequence these techniques are already being used, such as in the paper I just linked. > Being able to predict the next codon tells you zilch in terms of useable information. You have absolutely no way of knowing that apriori. And autogressive tasks can be more sophisticated than just next codon. > bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way Okay, feel free to think that. There's always this assumption of it "will never work on my field." I've done work on NLP and on proteins and read others' work on genetics. I think you will end up being surprised, although it might take a few years.
- alquemist 6y ago
  FWIW, transformers is to sequences what convnets is to grids, modulo important considerations like kernel size and normalization. Think of transformers as really wide (N) and really short (1) convolutions. Both are instances of graphnets with a suitable neighbor function. Once normalization was cracked by transformers, all sort of interesting graphnets became possible, though it's possible that stacked k-dimensional convolutions are sufficient in practice.
  - whimsicalism 6y ago
    I work in the field, I don't need the difference explained to me. > Think of transformers as really wide (N) and really short (1) convolutions Modern transformer networks are not "really short" and you're also conflating the difference between intra- and inter- attention. There is still a pitched battle being waged between convnets and transformers for sequences, although it looks like transformers have the upper hand accuracy wise right now, convnets are competitive speed-wise.
- Veedrac 6y ago
  > e: although AlphaFold appears to be convolutionally based! I suspect that'll change soon. “For the latest version of AlphaFold, used at CASP14, we created an attention-based neural network system” ?
the8472 6y ago
> but many were skeptical as to the longevity of the performance DeepMind was able to achieve For a non-biologist, on what is this skepticism based? Just purely based on following ML news it looks like the trend for ML solutions has been that they've overtaken expert-systems once they've gained a solid foodhold in a field. Maybe this is some perception bias. Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?
- whimsicalism 6y ago
  ML is a super overloaded term. There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.
  - penagwin 6y ago
    This. I believe technically just linear regression could be considered "machine learning".
    - misnome 6y ago
      I've seen people at bio conferences actively calling linear regression machine learning.
      - diab0lic 6y ago
        This is likely because linear regression meets most widely accepted definitions of machine learning. [0][1] It is simple and very effective when learning in linear space. [0] https://en.wikipedia.org/wiki/Machine_learning https://en.wikipedia.org/wiki/Machine_learning [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html
      - b3kart 6y ago
        Sorry, I don’t get it. Are you saying fitting a linear regression model to data and making predictions somehow isn’t machine learning? I am confused.
deleted 6y ago
[deleted]

whimsicalism 6y ago

Progress like this was, in my view, inevitable after the invention of unsupervised transformers. It'll be genetics next. e: although AlphaFold appears to be convolutionally based! I suspect that'll change soon.

klmr 6y ago
> It'll be genetics next. Which part of genetics are you thinking of? Much of genetics isn’t amenable to this kind of ML, because it isn’t some kind of optimisation problem. And many other parts don’t require ML because they can be modelled very closely using exact methods. ML does get used here, and sometimes to great effect (e.g. DeepVariant, which often outperforms other methods, but not by much — not because DeepVariant isn’t good, but rather because we have very efficient approximations to the exact solution).
- whimsicalism 6y ago
  What do you mean? Genetics is amenable because the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network. There are plenty of inferences that you would want to do on genetic sequences that we can't model exactly and there is some past work on doing stuff like this, although biology is usually a few years behind. https://www.nature.com/articles/s41592-018-0138-4 https://www.nature.com/articles/s41592-018-0138-4 e: for clarity
  - garmaine 6y ago
    This is word salad.
    - whimsicalism 6y ago
      Rude. I would appreciate substantive criticism, especially when I'm linking papers in Nature starting to do exactly what I'm talking about.
      - garmaine 6y ago
        I cannot give constructive feedback to something which is incomprehensible. "the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network" The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important. Language modeling in the context of machine learning typically means NLP methods. Genetics is nothing like natural language. Auto-regression is using (typically time series) information to predict the next codon. This makes very little sense in the context of genetics since, again, the genetic code is not an information carrying medium in the same sense as human language. Being able to predict the next codon tells you zilch in terms of useable information. "Depth of understanding by the network" ... what does that even mean??? The above sentence is a bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way. AKA word salad.
        whimsicalism 6y ago
        > The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important. aka a sequence. "a book is not a sequence so much as a discrete set of chapters which are themselves sequences of paragraphs which are themselves sequences of sentences" -> still a sequence these techniques are already being used, such as in the paper I just linked. > Being able to predict the next codon tells you zilch in terms of useable information. You have absolutely no way of knowing that apriori. And autogressive tasks can be more sophisticated than just next codon. > bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way Okay, feel free to think that. There's always this assumption of it "will never work on my field." I've done work on NLP and on proteins and read others' work on genetics. I think you will end up being surprised, although it might take a few years.
alquemist 6y ago
FWIW, transformers is to sequences what convnets is to grids, modulo important considerations like kernel size and normalization. Think of transformers as really wide (N) and really short (1) convolutions. Both are instances of graphnets with a suitable neighbor function. Once normalization was cracked by transformers, all sort of interesting graphnets became possible, though it's possible that stacked k-dimensional convolutions are sufficient in practice.
- whimsicalism 6y ago
  I work in the field, I don't need the difference explained to me. > Think of transformers as really wide (N) and really short (1) convolutions Modern transformer networks are not "really short" and you're also conflating the difference between intra- and inter- attention. There is still a pitched battle being waged between convnets and transformers for sequences, although it looks like transformers have the upper hand accuracy wise right now, convnets are competitive speed-wise.
Veedrac 6y ago
> e: although AlphaFold appears to be convolutionally based! I suspect that'll change soon. “For the latest version of AlphaFold, used at CASP14, we created an attention-based neural network system” ?

the8472 6y ago

> but many were skeptical as to the longevity of the performance DeepMind was able to achieve For a non-biologist, on what is this skepticism based? Just purely based on following ML news it looks like the trend for ML solutions has been that they've overtaken expert-systems once they've gained a solid foodhold in a field. Maybe this is some perception bias. Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?

whimsicalism 6y ago
ML is a super overloaded term. There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.
- penagwin 6y ago
  This. I believe technically just linear regression could be considered "machine learning".
  - misnome 6y ago
    I've seen people at bio conferences actively calling linear regression machine learning.
    - diab0lic 6y ago
      This is likely because linear regression meets most widely accepted definitions of machine learning. [0][1] It is simple and very effective when learning in linear space. [0] https://en.wikipedia.org/wiki/Machine_learning https://en.wikipedia.org/wiki/Machine_learning [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html
    - b3kart 6y ago
      Sorry, I don’t get it. Are you saying fitting a linear regression model to data and making predictions somehow isn’t machine learning? I am confused.

deleted 6y ago

[deleted]