19 ms·

> I feel like its important to find a topic where your reasonably confident that there is large margins of potential improvement due to something like lack of e

> I feel like its important to find a topic where your reasonably confident that there is large margins of potential improvement due to something like lack of exploration or focus by other people. I don't have a PhD so I can't comment first hand on the experience, but one thing I remember from a conversation with the head of a research institute at a large state university, is that there isn't really a lot of unexplored ground. Modern research is incremental. And those tiny discoveries multiplied by thousands of PhD researchers gradually expand our overall body of knowledge. I hope that doesnt come across as discouraging. It's just that it stuck with me for some reason.

auggierose 5y ago
There are plenty of big advancements to be made. It's just because of the way research funding works, nobody has the funds or the time to work on them, because, yeah, these advancements are not easy or quick or predictable to come by. Sounds to me like the head of a research institute rationalising to himself why the research coming out of his institute is basically worthless.
- native_samples 5y ago
  I think the story that rammed that home to me is this one, which is, ironically, about a stagnant (but important!) scientific field being totally wiped out by Google in the space of just 4 years by applying AI to the problem: https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp13-what-just-happened/ https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp... https://moalquraishi.wordpress.com/2020/12/08/alphafold2-casp14-it-feels-like-ones-child-has-left-home/ https://moalquraishi.wordpress.com/2020/12/08/alphafold2-cas... Yes, progress was (not exactly) coming through tons of tiny incremental advances, right up until an industrial lab got involved - as a side project no less - at which point the academics were blown out of the water.
  - dekhn 5y ago
    that's an overstatement of DeepMind's results and not fair to the people in the field.
    - native_samples 5y ago
      I believe what I wrote is a fair summary of the blog posts, which were written by someone in the field who competed at CASP.
      - dekhn 5y ago
        I've also competed in CASP. His posts are really misleading.
        native_samples 5y ago
        I'd be really interested in knowing where you disagree and why, if you have time to do a writeup.
        dekhn 5y ago
        There's not much to say. DM made a significant improvement to homology-based modelling using some very recent technology and fair amount of CPU power. They collaborate with a long-time CASP player (https://en.wikipedia.org/wiki/David_Tudor_Jones https://en.wikipedia.org/wiki/David_Tudor_Jones) who has been doing leading edge protein folding for decades. And CASP is a competition, which DM excels at. Finally, they picked the "easy" category of CASP, for which there is a huge amount of side data which ultimately, when properly processed, provides structural constraint information good enough to produce very high quality structures. By not addressing ab initio protein folding, they didn't solve the protein folding problem, they solved the "homologous protein structure prediction problem", which is far less grand. I did CASP back in 2001 or 2002 using alignment methods and protein threading, nothing special (we did terribly). But I told everybody at the conference that ML would eventually surpass the best human predictions, and people said (correctly) that at the time: we didn't have enough data (3D structures) to train on, we didn't have algorithms that were any good (practical backprop for deep NNs, embeddings, transformers), and there wasn't enough CPU time. The data problem was addressed (by having huge numbers of sequence alignments for most protein structures), the algorithms have obviously gotten better, especially transformers, and DeepMind has a lot of CPU (well, TPU). So basically my prediction came true when the underlying restrictions were removed (which to me seems "obvious"). The academic community would have done this two years after DM if DM hadn't done it (although their wins probably would be much smaller). DM just got there faster, by exploiting a number of advantages. None of this really moves anything forward. Being able to predict protein structures of homologous proteins is not the grand challenge problem, and what DM did tells us nothing about the physics of folding. And it doesn't produce results good enough to do structure based drug design. Really, they should have just put out a PR that said they did well in CASP and would be open sourcing the model, the training data, and a trained model, and then done that six months later.
activatedgeek 5y ago
> Modern research is incremental. All research is incremental. Even though in context it appears to speak to the right idea, I just want to make sure the casual reader doesn't emphasize too much on the "modern" part. This idea of groundbreaking research happening out-of-the-blue is nothing but a pop-sci fantasy. Every groundbreaking idea, whether it be Newton/Leibniz's calculus, Marx's manifesto, or more recently the deep learning revolution led by Bengio/Hinton/LeCun/Schmidhuber, are only groundbreaking in hindsight. Ideas take time to come into the shape that appeals to the spirit of the times. Sometimes, they win a lottery (e.g. the Hardware lottery [1]). [1]: https://hardwarelottery.github.io https://hardwarelottery.github.io
aqsalose 5y ago
>is that there isn't really a lot of unexplored ground My take is, there isn't a lot of obvious unexplored ground. Where "obvious" includes many things fresh graduate student would come up thinking about some logical first steps. Possibly also many things what most professors would consider. However, that does not say anything about unobvious unexplored ground. Also, many professors seem to think about in paper-sized increments. Narratives about blessings of incremental steps to expand the body of knowledge sound suspiciously self-serving to researches who publish incremental steps. (And given the noise in ML experiments, are the incremental steps incremental or merely steps in random directions?) [1] One way I have started to think about this: How many of the results one uses everyday are product of small increments? How small the increments were? Is another study really helpful to finding the next the such result? Naive case for anti-incrementalist science is naturally false, as every result is built on something, but I wonder if the prominent narrative is discounting the role of genuine novel insights and ideas to the detriment of science, as people do not try to obtain them. The question "how" is above my paygrade (FYI, I am no longer in academia), but I suspect one place where useful things often have came from people who manage to ask new but interesting and useful questions, which open up the much desired new unexplored territory. Which are good questions nobody in the field seems to ask? [1] Wasn't there a paper some years back how most of the advances and tricks in neural net structures turned out not very important nor generalizable compared to more traditional models given more compute? Can't find the reference now.

auggierose 5y ago

There are plenty of big advancements to be made. It's just because of the way research funding works, nobody has the funds or the time to work on them, because, yeah, these advancements are not easy or quick or predictable to come by. Sounds to me like the head of a research institute rationalising to himself why the research coming out of his institute is basically worthless.

native_samples 5y ago
I think the story that rammed that home to me is this one, which is, ironically, about a stagnant (but important!) scientific field being totally wiped out by Google in the space of just 4 years by applying AI to the problem: https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp13-what-just-happened/ https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp... https://moalquraishi.wordpress.com/2020/12/08/alphafold2-casp14-it-feels-like-ones-child-has-left-home/ https://moalquraishi.wordpress.com/2020/12/08/alphafold2-cas... Yes, progress was (not exactly) coming through tons of tiny incremental advances, right up until an industrial lab got involved - as a side project no less - at which point the academics were blown out of the water.
- dekhn 5y ago
  that's an overstatement of DeepMind's results and not fair to the people in the field.
  - native_samples 5y ago
    I believe what I wrote is a fair summary of the blog posts, which were written by someone in the field who competed at CASP.
    - dekhn 5y ago
      I've also competed in CASP. His posts are really misleading.
      - native_samples 5y ago
        I'd be really interested in knowing where you disagree and why, if you have time to do a writeup.
        dekhn 5y ago
        There's not much to say. DM made a significant improvement to homology-based modelling using some very recent technology and fair amount of CPU power. They collaborate with a long-time CASP player (https://en.wikipedia.org/wiki/David_Tudor_Jones https://en.wikipedia.org/wiki/David_Tudor_Jones) who has been doing leading edge protein folding for decades. And CASP is a competition, which DM excels at. Finally, they picked the "easy" category of CASP, for which there is a huge amount of side data which ultimately, when properly processed, provides structural constraint information good enough to produce very high quality structures. By not addressing ab initio protein folding, they didn't solve the protein folding problem, they solved the "homologous protein structure prediction problem", which is far less grand. I did CASP back in 2001 or 2002 using alignment methods and protein threading, nothing special (we did terribly). But I told everybody at the conference that ML would eventually surpass the best human predictions, and people said (correctly) that at the time: we didn't have enough data (3D structures) to train on, we didn't have algorithms that were any good (practical backprop for deep NNs, embeddings, transformers), and there wasn't enough CPU time. The data problem was addressed (by having huge numbers of sequence alignments for most protein structures), the algorithms have obviously gotten better, especially transformers, and DeepMind has a lot of CPU (well, TPU). So basically my prediction came true when the underlying restrictions were removed (which to me seems "obvious"). The academic community would have done this two years after DM if DM hadn't done it (although their wins probably would be much smaller). DM just got there faster, by exploiting a number of advantages. None of this really moves anything forward. Being able to predict protein structures of homologous proteins is not the grand challenge problem, and what DM did tells us nothing about the physics of folding. And it doesn't produce results good enough to do structure based drug design. Really, they should have just put out a PR that said they did well in CASP and would be open sourcing the model, the training data, and a trained model, and then done that six months later.

activatedgeek 5y ago

> Modern research is incremental. All research is incremental. Even though in context it appears to speak to the right idea, I just want to make sure the casual reader doesn't emphasize too much on the "modern" part. This idea of groundbreaking research happening out-of-the-blue is nothing but a pop-sci fantasy. Every groundbreaking idea, whether it be Newton/Leibniz's calculus, Marx's manifesto, or more recently the deep learning revolution led by Bengio/Hinton/LeCun/Schmidhuber, are only groundbreaking in hindsight. Ideas take time to come into the shape that appeals to the spirit of the times. Sometimes, they win a lottery (e.g. the Hardware lottery [1]). [1]: https://hardwarelottery.github.io https://hardwarelottery.github.io

aqsalose 5y ago

>is that there isn't really a lot of unexplored ground My take is, there isn't a lot of obvious unexplored ground. Where "obvious" includes many things fresh graduate student would come up thinking about some logical first steps. Possibly also many things what most professors would consider. However, that does not say anything about unobvious unexplored ground. Also, many professors seem to think about in paper-sized increments. Narratives about blessings of incremental steps to expand the body of knowledge sound suspiciously self-serving to researches who publish incremental steps. (And given the noise in ML experiments, are the incremental steps incremental or merely steps in random directions?) [1] One way I have started to think about this: How many of the results one uses everyday are product of small increments? How small the increments were? Is another study really helpful to finding the next the such result? Naive case for anti-incrementalist science is naturally false, as every result is built on something, but I wonder if the prominent narrative is discounting the role of genuine novel insights and ideas to the detriment of science, as people do not try to obtain them. The question "how" is above my paygrade (FYI, I am no longer in academia), but I suspect one place where useful things often have came from people who manage to ask new but interesting and useful questions, which open up the much desired new unexplored territory. Which are good questions nobody in the field seems to ask? [1] Wasn't there a paper some years back how most of the advances and tricks in neural net structures turned out not very important nor generalizable compared to more traditional models given more compute? Can't find the reference now.