3 ms·

Unfortunately while this advice sounds useful, it isn’t. It might be useful if you measure impact is the fake metrics like citation count but if you want to mea

Unfortunately while this advice sounds useful, it isn’t. It might be useful if you measure impact is the fake metrics like citation count but if you want to measure with actual tangible impact on the real world, you have a rude awakening before you. To me there are 2 ways a paper can create impact: 1. The paper represents such an impressive leap in performance over existing methods in AI, that it is obviously impactful. Unfortunately, this way of generating impact is dominated by industry. No one can expect Academia to train O1, SAM, GPT5 etc. AI rewards scale, scale requires money, resources and manpower and Academia has none. In the early days of AI, there were rare moments when this was possible, AlexNet, Adam, Transformers, PPO etc. Is it still possible? I do not know, I have not seen anything in the last 3 years and I’m not optimistic many such opportunities are left. Even validating your idea tends to require the scale of industry. 2. The paper affects the thought process of other AI researchers and thus you are indirectly impactful if any of them cause big leaps in AI performance. Unfortunately here is where Academia has shot itself in the foot by generating so many damn papers every year (>10,000). There are just so many, that the effect of any 1 paper is meaningless. In fact the only way to be impactful now is to be in a social circle of great researchers, so that you know your social circle will read your paper and later if any of them make big performance improvements, you can believe that you played a small role in it. I have spoken to a lot of ML researchers, and they told me they choose papers to read just based on people and research groups they know. Even being a NeurIPS spotlight paper, means less than 10% of researchers will read your paper, maybe it will go to 50% if it’s a NEURIPS best paper but even that I doubt. How many researchers remember last year’s NEURIPS best paper? The only solution to problem 2, is radical. The ML community needs to come together and limit the number of papers it wide releases. Let us say it came out and said that yearly only 20 curated papers will be widely published. Then you can bet most of the ML community will read all 20 of those papers and engage with it deeply as they will be capable of spending more than a day at least thinking about the paper. Of course you can still publish on arxiv, share with friends etc but unless such a dramatic cutdown is made I don’t see how you can be an actually impactful AI researcher in Academia when option 1 is too expensive and option 2 is made impossible.

EnigmaFlare 2y ago
If you were a really good academic, you could come up with a theory that predicts the performance of a model you haven't actually tested. Physics is full of this - theories that predict something novel followed by other people doing expensive experiments to test them. The guy who published the theory gets the credit. That's probably what we want from academic AI researchers - theories that we can eventually use to design models, rather than just haphazardly building something and wondering how good it will be.
- light_hue_1 2y ago
  That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried. Just because that's what people want, doesn't mean we can produce it. I often talk to funding agencies about things like this. "We don't want to fund boring research, only what will give us the ultimate theory of how everything works". That's not how science or progress work.
  - aleph_minus_one 2y ago
    > That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried. In my opinion the problem rather is that considering the current AI gold rush, people are rather eager to throw newly implemented models around instead of thinking really deeply how an insanely better model could look like.
SonOfLilit 2y ago
With one outlier, by the one academic AI lab that continues to produce impactful research every year, Christopher Ré's at Stanford (although the raised VC money now so who knows), FlashAttention.
bonoboTP 2y ago
> Academia has shot itself in the foot by generating so many damn papers every year (>10,000). Each PhD student needs 3-5 papers to graduate. Nowadays you even need 1-2 to simply get into a PhD program. The huge number of papers compared to, say, 10 or 15 years ago simply reflects how many more PhD students there are in this area. It definitely isn't sustainable but academia is a distributed system. There's no way to coordinate publishing less. No individual agent will rationally choose that. Especially that it's not just competition within AI but other CS fields too for huge grants and the grant agencies defer to citation and publication metrics. CVPR is among the top scientific publication venues by various metrics and this state of affairs also benefits the field. There are reasons for why things are this way but the way it's going is clearly unsustainable. It's a red queen race, similar to grade inflation, credential inflation, and various forms of cost disease.

EnigmaFlare 2y ago

If you were a really good academic, you could come up with a theory that predicts the performance of a model you haven't actually tested. Physics is full of this - theories that predict something novel followed by other people doing expensive experiments to test them. The guy who published the theory gets the credit. That's probably what we want from academic AI researchers - theories that we can eventually use to design models, rather than just haphazardly building something and wondering how good it will be.

light_hue_1 2y ago
That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried. Just because that's what people want, doesn't mean we can produce it. I often talk to funding agencies about things like this. "We don't want to fund boring research, only what will give us the ultimate theory of how everything works". That's not how science or progress work.
- aleph_minus_one 2y ago
  > That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried. In my opinion the problem rather is that considering the current AI gold rush, people are rather eager to throw newly implemented models around instead of thinking really deeply how an insanely better model could look like.

SonOfLilit 2y ago

With one outlier, by the one academic AI lab that continues to produce impactful research every year, Christopher Ré's at Stanford (although the raised VC money now so who knows), FlashAttention.

bonoboTP 2y ago

> Academia has shot itself in the foot by generating so many damn papers every year (>10,000). Each PhD student needs 3-5 papers to graduate. Nowadays you even need 1-2 to simply get into a PhD program. The huge number of papers compared to, say, 10 or 15 years ago simply reflects how many more PhD students there are in this area. It definitely isn't sustainable but academia is a distributed system. There's no way to coordinate publishing less. No individual agent will rationally choose that. Especially that it's not just competition within AI but other CS fields too for huge grants and the grant agencies defer to citation and publication metrics. CVPR is among the top scientific publication venues by various metrics and this state of affairs also benefits the field. There are reasons for why things are this way but the way it's going is clearly unsustainable. It's a red queen race, similar to grade inflation, credential inflation, and various forms of cost disease.