18 ms·

I've started my ml/dl focused PhD a short while back, and I'm beginning to realize just how difficult it is to achieve significant and as well as consistent res

I've started my ml/dl focused PhD a short while back, and I'm beginning to realize just how difficult it is to achieve significant and as well as consistent results. Actual improvements that come along after multiple rounds of experimentation sometimes only amount to a few percentage points of improvement, and after trying it on a new dataset or new randomization seeds and even that could disappear. Sometimes i'm afraid of running too many experiments since I feel that could make me implicit to academic fraud and it might be better just not to know lol. Resource constraint also seems to be a serious thing, looking at a full list of ablation experiments, together with multiple runs to obtain error bars, could well take up to a week. If any of the experiments reveal bad results it could also mean back to the drawing board. It seems one would need multiple readily available GPUs in order to be able to really experiment at a comfortable pace. I also wonder if choice of topic has an important part, since I was sort of assigned my topic which I just decided to do my best to see through. I feel like its important to find a topic where your reasonably confident that there is large margins of potential improvement due to something like lack of exploration or focus by other people. Similarly it might be a good idea to bail if simple initial experimentation yields bad results, or if you find yourself wrangling with increasingly complicated methods for long times but still no obvious signs of progress. I'd love to hear some others chip in with their experiences on how best to manage the research process cause I feel like I haven't quite gotten the catch of it yet. Are there any good tricks (in the strategy sense, not fraud) that could make the process easier?

throwawayboise 5y ago
> I feel like its important to find a topic where your reasonably confident that there is large margins of potential improvement due to something like lack of exploration or focus by other people. I don't have a PhD so I can't comment first hand on the experience, but one thing I remember from a conversation with the head of a research institute at a large state university, is that there isn't really a lot of unexplored ground. Modern research is incremental. And those tiny discoveries multiplied by thousands of PhD researchers gradually expand our overall body of knowledge. I hope that doesnt come across as discouraging. It's just that it stuck with me for some reason.
- auggierose 5y ago
  There are plenty of big advancements to be made. It's just because of the way research funding works, nobody has the funds or the time to work on them, because, yeah, these advancements are not easy or quick or predictable to come by. Sounds to me like the head of a research institute rationalising to himself why the research coming out of his institute is basically worthless.
  - native_samples 5y ago
    I think the story that rammed that home to me is this one, which is, ironically, about a stagnant (but important!) scientific field being totally wiped out by Google in the space of just 4 years by applying AI to the problem: https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp13-what-just-happened/ https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp... https://moalquraishi.wordpress.com/2020/12/08/alphafold2-casp14-it-feels-like-ones-child-has-left-home/ https://moalquraishi.wordpress.com/2020/12/08/alphafold2-cas... Yes, progress was (not exactly) coming through tons of tiny incremental advances, right up until an industrial lab got involved - as a side project no less - at which point the academics were blown out of the water.
    - dekhn 5y ago
      that's an overstatement of DeepMind's results and not fair to the people in the field.
      - native_samples 5y ago
        I believe what I wrote is a fair summary of the blog posts, which were written by someone in the field who competed at CASP.
        dekhn 5y ago
        I've also competed in CASP. His posts are really misleading.
henrikeh 5y ago
I am about at the end of my PhD* (a few months left) and my supervisor's advice still rings true: try to keep the feedback loops short. If you have to spend months figuring out if something is worth doing, then it becomes rather risky to try. However, if you can shorten the cycle from idea to rough confirmation down to a week or two, then you can afford to explore the options. The trick is of course that there are several ways to approach this: you can develop more flexible tools, become more independent in the lab (if you have lab work), learn more theory, learn more tools, buy more compute power, pick easy problems etc. Another angle I have thought about (and learned the hard way), is that there is a big difference between research and development. My topic is closely related to engineering and thus it is sometimes attractive to want to "solve a problem". Unfortunately, many measures of "good research" supposes a relevant application, but "solving a problem" is much more difficult than "understanding a problem". The understanding truly has to come first. Also, these two pieces of advice (Short learning feedback; focus on understanding not solving), are not guarantees to produce "groundbreaking" research. In the end, the most important product of a PhD education is you and the skills you develop. * A PhD in Denmark is a bit special, in that it is very short (essentially limited to 3 years) and thus has a slightly different scope.
- codethief 5y ago
  > * A PhD in Denmark is a bit special, in that it is very short (essentially limited to 3 years) and thus has a slightly different scope. I think 3-4 years is the norm in most countries where you do a (two-years) master's before you do a PhD.
- ackbar03 5y ago
  thanks for chipping in > try to keep the feedback loops short Yes, just as it is almost mantra for startups, I agree that this is extremely important as well from my not so smooth experience so far. I think its also what makes DL kind of hard as well, experiments can take a long time to run depending on access to hardware.

throwawayboise 5y ago

> I feel like its important to find a topic where your reasonably confident that there is large margins of potential improvement due to something like lack of exploration or focus by other people. I don't have a PhD so I can't comment first hand on the experience, but one thing I remember from a conversation with the head of a research institute at a large state university, is that there isn't really a lot of unexplored ground. Modern research is incremental. And those tiny discoveries multiplied by thousands of PhD researchers gradually expand our overall body of knowledge. I hope that doesnt come across as discouraging. It's just that it stuck with me for some reason.

auggierose 5y ago
There are plenty of big advancements to be made. It's just because of the way research funding works, nobody has the funds or the time to work on them, because, yeah, these advancements are not easy or quick or predictable to come by. Sounds to me like the head of a research institute rationalising to himself why the research coming out of his institute is basically worthless.
- native_samples 5y ago
  I think the story that rammed that home to me is this one, which is, ironically, about a stagnant (but important!) scientific field being totally wiped out by Google in the space of just 4 years by applying AI to the problem: https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp13-what-just-happened/ https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp... https://moalquraishi.wordpress.com/2020/12/08/alphafold2-casp14-it-feels-like-ones-child-has-left-home/ https://moalquraishi.wordpress.com/2020/12/08/alphafold2-cas... Yes, progress was (not exactly) coming through tons of tiny incremental advances, right up until an industrial lab got involved - as a side project no less - at which point the academics were blown out of the water.
  - dekhn 5y ago
    that's an overstatement of DeepMind's results and not fair to the people in the field.
    - native_samples 5y ago
      I believe what I wrote is a fair summary of the blog posts, which were written by someone in the field who competed at CASP.
      - dekhn 5y ago
        I've also competed in CASP. His posts are really misleading.

henrikeh 5y ago

I am about at the end of my PhD* (a few months left) and my supervisor's advice still rings true: try to keep the feedback loops short. If you have to spend months figuring out if something is worth doing, then it becomes rather risky to try. However, if you can shorten the cycle from idea to rough confirmation down to a week or two, then you can afford to explore the options. The trick is of course that there are several ways to approach this: you can develop more flexible tools, become more independent in the lab (if you have lab work), learn more theory, learn more tools, buy more compute power, pick easy problems etc. Another angle I have thought about (and learned the hard way), is that there is a big difference between research and development. My topic is closely related to engineering and thus it is sometimes attractive to want to "solve a problem". Unfortunately, many measures of "good research" supposes a relevant application, but "solving a problem" is much more difficult than "understanding a problem". The understanding truly has to come first. Also, these two pieces of advice (Short learning feedback; focus on understanding not solving), are not guarantees to produce "groundbreaking" research. In the end, the most important product of a PhD education is you and the skills you develop. * A PhD in Denmark is a bit special, in that it is very short (essentially limited to 3 years) and thus has a slightly different scope.

codethief 5y ago
> * A PhD in Denmark is a bit special, in that it is very short (essentially limited to 3 years) and thus has a slightly different scope. I think 3-4 years is the norm in most countries where you do a (two-years) master's before you do a PhD.
ackbar03 5y ago
thanks for chipping in > try to keep the feedback loops short Yes, just as it is almost mantra for startups, I agree that this is extremely important as well from my not so smooth experience so far. I think its also what makes DL kind of hard as well, experiments can take a long time to run depending on access to hardware.