6 ms·

Math is such an interesting field. People can work for decades and not make progress, then discover something in a moment of clarity from some seemingly unrelat

Math is such an interesting field. People can work for decades and not make progress, then discover something in a moment of clarity from some seemingly unrelated problem. As a programmer, I don't have that type of patience.

groby_b 2y ago
Get yourself a slower compiler ;) But all kidding aside, you likely will have those moments. Not because you had patience, but because over you career you collected enough knowledge bits in your brain that they'll at some point click together in extremely odd shapes. Like with maths, if you choose to follow up, this is either a moment of clarity, or the moment you enter crankdom.
godelski 2y ago
Honestly, this happens in a lot of fields, including programming. I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesting. I hear you, we should be efficient and not waste money. But is there actually good evidence that we have strong predictive powers here? There's at least strong evidence that dark horses are quite common in the innovation space and exceptionally common in major breakthroughs. So even if we want to primarily focus on funding promising directions there is still good evidence that optimal funding requires funding unpopular ideas. Which if we think about a lot of this, it should make sense anyways. Just by thinking about optimization theory. It is often quite good to add noise so that your optimization function can escape local minima. We can only travel directly to the global optima if we know exactly where it is. But should we not expect that expert predictive power is much better at pointing to local optima rather than global? (global very likely doesn't exist but that doesn't mean there aren't better optima). There's also many famous scientists that didn't "spend much time working." I add quotes, because if you're a researcher you'd naturally understand there's no real thing as "not working." There is only active work and inactive work. You're likely consumed by the topics and problems you're trying to solve. So doing things like going on walks, playing your favorite sport, or whatever ends up being beneficial as you can relax and shift between focused and creative modes. But that doesn't happen as much if your boss thinks "working" is staring at the chalkboard. Sometimes it is best to go sit outside and daydream, while other times it is best to hammer your head against that metaphorical chalkboard. > I don't have that type of patience. As for this, patience is a skill. Delayed rewards. Long term rewards are often noisier and more difficult to attribute to their appropriate causes. Even if the long term rewards are substantially greater than the short term, and the timeframe isn't too large, most people prefer the short term. Not just because reward, but because interpretability. As an example, just think about education in of itself. Lots of effort but also lots of reward. Even though the process is very noisy and it is unclear which aspects of education contributed the most to success, it is very clear that there's a strong connection between education and success (does not mean there aren't other pathways to success nor that you are guaranteed to be successful by being educated. It can be easy to conflate these things).
- TeMPOraL 2y ago
  > I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesting. I honestly feel this is mostly because of a (possibly well-justified) fear that, were we to not do that, almost all the money would quickly get captured by grifters whose ideas are fake and of no research value, but who were best at convincing they're worth funding. We can't read peoples' hearts, so we need to force out some output we can discriminate on.
  - godelski 2y ago
    I believe you are correct, but at the same time, are we not already giving a lot of money to grifters? For most research funding, I think the incentives are misaligned for the type of grifters we're trying to prevent, i.e. people just pocketing the money. The money is too low and the barriers are too high. The grifting that does happen is more about metric hacking and is about the status one achieves in academia. Yeah, there's high profile people that make good money on this, but at the same time many in the respective fields are not shocked to find that their papers were faked. There were usually existing accusations. But the structures in place are not well aligned to oust or investigate them. The publish or perish paradigm gives little time for replication experiments and you should come with strong evidence before making accusations. We're also getting better at this, but more awareness and more discovery has not led to these people losing their positions our status. (We even see universities protect them at times) Take this recent plagiarism case[0]. Most authors have quite good citation records and metrics, but how does a work that's so egregious not cause other works to be investigated, it warrants suspicion. We have network graphs of researchers, with their collaborators, but why do we not build these graphs for abusers? Most of the time the abuse is purged behind closed doors and so it even makes it harder to build the networks. CS is especially egregious for this. I lost a lot of respect for a bunch of my ML researcher peers when they promoted Rabbit. The demo was so egregiously faked and none of the claims made sense given the state of research. But we throw tons of money at these types of problems. We thrive on hype bubbles and don't purge the conmen. Hell, we often reward bad behavior. Like how that intern that got fired from Bytedance for manipulating code won Best Paper and NeurIPS[1]. The HumanEval paper is ridiculously naive (to think you can write "leetcode style problems" and that because you wrote them "by hand" that they won't be spoiled by training on GitHub. You can find identical code to most of the canonical solutions trivially!). There's tons of memes in ML around plagiarism and as best as I can tell all these authors are still doing just fine. How an egregious case doesn't result in a 1 year probation for a conference is beyond me. We suspend students for this academic misconduct but bury it for professors and grad students? The incentives are all wrong and we need to take a serious look at ourselves. Because we are in fact protecting the grifters. It is just lucky that most people are uninterested in grifting in this domain, though as said, there isn't much incentive to (obviously changing with the prominence in ML). [0] https://openreview.net/forum?id=cIKQp84vqN https://openreview.net/forum?id=cIKQp84vqN [1] https://www.wired.com/story/bytedance-intern-best-paper-neurips/ https://www.wired.com/story/bytedance-intern-best-paper-neur...

groby_b 2y ago

Get yourself a slower compiler ;) But all kidding aside, you likely will have those moments. Not because you had patience, but because over you career you collected enough knowledge bits in your brain that they'll at some point click together in extremely odd shapes. Like with maths, if you choose to follow up, this is either a moment of clarity, or the moment you enter crankdom.

godelski 2y ago

Honestly, this happens in a lot of fields, including programming. I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesting. I hear you, we should be efficient and not waste money. But is there actually good evidence that we have strong predictive powers here? There's at least strong evidence that dark horses are quite common in the innovation space and exceptionally common in major breakthroughs. So even if we want to primarily focus on funding promising directions there is still good evidence that optimal funding requires funding unpopular ideas. Which if we think about a lot of this, it should make sense anyways. Just by thinking about optimization theory. It is often quite good to add noise so that your optimization function can escape local minima. We can only travel directly to the global optima if we know exactly where it is. But should we not expect that expert predictive power is much better at pointing to local optima rather than global? (global very likely doesn't exist but that doesn't mean there aren't better optima). There's also many famous scientists that didn't "spend much time working." I add quotes, because if you're a researcher you'd naturally understand there's no real thing as "not working." There is only active work and inactive work. You're likely consumed by the topics and problems you're trying to solve. So doing things like going on walks, playing your favorite sport, or whatever ends up being beneficial as you can relax and shift between focused and creative modes. But that doesn't happen as much if your boss thinks "working" is staring at the chalkboard. Sometimes it is best to go sit outside and daydream, while other times it is best to hammer your head against that metaphorical chalkboard. > I don't have that type of patience. As for this, patience is a skill. Delayed rewards. Long term rewards are often noisier and more difficult to attribute to their appropriate causes. Even if the long term rewards are substantially greater than the short term, and the timeframe isn't too large, most people prefer the short term. Not just because reward, but because interpretability. As an example, just think about education in of itself. Lots of effort but also lots of reward. Even though the process is very noisy and it is unclear which aspects of education contributed the most to success, it is very clear that there's a strong connection between education and success (does not mean there aren't other pathways to success nor that you are guaranteed to be successful by being educated. It can be easy to conflate these things).

TeMPOraL 2y ago
> I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesting. I honestly feel this is mostly because of a (possibly well-justified) fear that, were we to not do that, almost all the money would quickly get captured by grifters whose ideas are fake and of no research value, but who were best at convincing they're worth funding. We can't read peoples' hearts, so we need to force out some output we can discriminate on.
- godelski 2y ago
  I believe you are correct, but at the same time, are we not already giving a lot of money to grifters? For most research funding, I think the incentives are misaligned for the type of grifters we're trying to prevent, i.e. people just pocketing the money. The money is too low and the barriers are too high. The grifting that does happen is more about metric hacking and is about the status one achieves in academia. Yeah, there's high profile people that make good money on this, but at the same time many in the respective fields are not shocked to find that their papers were faked. There were usually existing accusations. But the structures in place are not well aligned to oust or investigate them. The publish or perish paradigm gives little time for replication experiments and you should come with strong evidence before making accusations. We're also getting better at this, but more awareness and more discovery has not led to these people losing their positions our status. (We even see universities protect them at times) Take this recent plagiarism case[0]. Most authors have quite good citation records and metrics, but how does a work that's so egregious not cause other works to be investigated, it warrants suspicion. We have network graphs of researchers, with their collaborators, but why do we not build these graphs for abusers? Most of the time the abuse is purged behind closed doors and so it even makes it harder to build the networks. CS is especially egregious for this. I lost a lot of respect for a bunch of my ML researcher peers when they promoted Rabbit. The demo was so egregiously faked and none of the claims made sense given the state of research. But we throw tons of money at these types of problems. We thrive on hype bubbles and don't purge the conmen. Hell, we often reward bad behavior. Like how that intern that got fired from Bytedance for manipulating code won Best Paper and NeurIPS[1]. The HumanEval paper is ridiculously naive (to think you can write "leetcode style problems" and that because you wrote them "by hand" that they won't be spoiled by training on GitHub. You can find identical code to most of the canonical solutions trivially!). There's tons of memes in ML around plagiarism and as best as I can tell all these authors are still doing just fine. How an egregious case doesn't result in a 1 year probation for a conference is beyond me. We suspend students for this academic misconduct but bury it for professors and grad students? The incentives are all wrong and we need to take a serious look at ourselves. Because we are in fact protecting the grifters. It is just lucky that most people are uninterested in grifting in this domain, though as said, there isn't much incentive to (obviously changing with the prominence in ML). [0] https://openreview.net/forum?id=cIKQp84vqN https://openreview.net/forum?id=cIKQp84vqN [1] https://www.wired.com/story/bytedance-intern-best-paper-neurips/ https://www.wired.com/story/bytedance-intern-best-paper-neur...