3 ms·

I like the article directionally but fear the examples are too short-sighted for most AI researchers. Picking something useful in 1-2 years is a reason to go t

I like the article directionally but fear the examples are too short-sighted for most AI researchers. Picking something useful in 1-2 years is a reason to go to industry, not research, and leads to mostly incremental units that if you don't do, someone else will. Yes, hot topics are good because they signal a time of fertile innovation. But not if your vision is so shallow that you will have half of IBM, Google, and YC competing with you before you start or by the time of your first publication (6-12mo). If you are a top student, well-educated already, with top resources and your own mentees, and your advisor is an industry leader who already knows where your work will go, maybe go to the thickest 1-2 year out AI VC fest, but that's not most PhD students. A 'practical' area would be obvious to everyone in 5 years, but winnowing out the crowd, there should not be much point to it today nor 1-2 years without something fundamentally changing. It should be tangible enough to be relevant and enticing, but too expensive for whatever reasons. More fundamental research would be even more years out. This gives you a year or two to dig into the problem, and another year or two to build out fundamental solutions, and then a couple years of cranking. From there, rinse-and-repeat via your own career or those of your future students. Some of my favorite work took 1-2 years of research to establish the problem, not just the solutions. Two of the projects here were weird at first as problems on a longer time scale, but ended up as part of $20M grant and software many folks here use & love, and another, a 10 year test of time award. (And another, arguably a $100M+ division at Nvidia). In contrast, most of my topic-of-the-year stuff didn't matter and was interchangeable with work by others. Edit: The speech by Hamming on "You and your research" hits on similar themes and speaks more to my experiences here: https://fs.blog/great-talks/richard-hamming-your-research/ https://fs.blog/great-talks/richard-hamming-your-research/

QuadmasterXLII 2y ago
If no one did the research that anyone could do, it would never get done.
- marcosdumay 2y ago
  In most markets, it pays to be contrarian. ... and when everybody decides to be a contrarian, the contrarian position is to do the popular thing. But the catch is that this never happens on practice.
- lmeyerov 2y ago
  As I wrote, different folks will already be chasing more immediate projects as they're too attractive: Think masters students, industry labs, startups, etc. Rules are meant to be broken. For example, a professor will often put a masters student on a cute me-too one-off project in a hot area as the short-termism helps the student have a smoother time, and the professor is trading expected low impact in the the project for a cheap shot to see if there is a more interesting question behind the immediate one. Much of the PhD is about learning to spot & navigate problems, not solutions. Young PhD students can totally pick the same me-too problems of masters students. The advisors would likely just think the student is using the same approach to seeing where the PhD worthy problem is. If still nothing, should be ready to drop the topic, or if they truly love me-too work more than the PhD process, leave for a better environment for that, like a company / industry lab / startup throwing more resources at the problem. Also, not every phd needs to do the original topic of attempting impact. You can totally use the PhD years to learn how to do me-too & unimpactful work or even the most weird & niche problems: as long as at least one advisor thinks they can stay funded while supporting you, most programs will let you through.
- bonoboTP 2y ago
  You're right, but the individual has to ask if entering a crowded niche is worth it or not. Does your main strength lie in absurd workload to grind out the paper a few months/weeks earlier to Arxiv than the next group will? The risk reward calculus still often benefits this approach. The risk is that you get scooped. If you do something nonobvious, the risk is that it doesn't pan out or the reviewers don't get the point. The latter is harder to solve by grinding. So many phds will chose the former, as at least putting in ridiculous hours feels more actionable. Also, doing the straightforward thing and getting scooped is less ego-hurting than failing at your own special pet idea. I think many who aren't in grad school don't understand how extremely important it is to have publications. At the beginning you must get publications to do internships and to build contacts at conferences. Then later it becomes imperative in order to graduate and get your next position. Nowadays you must even have multiple top conference papers just to start a PhD. Of course from the point of view of the advisor with 15 PhD students, the ideal student would do high risk high reward work and if 3 projects become super successful, 5 become quite successful and 7 yield nothing, that's still great for the advisor. But the 7 who lost several years would rather not go through this.
bonoboTP 2y ago
I find that in AI / ML / computer vision, the thinking horizon has shrunk a lot. Reviewers prize straightforward incremental work that nevertheless beats the SOTA. New problem formulations are too hard to decipher and understand deeply and nobody has time nowadays. Everything is super sped up. There's a fire hose of papers coming out that all do slightly different things, or the same concurrently. In this area even the best, most respected professors don't have long term laid out research plans. Grant plans must exist but they never quite pan out that way. Researchers in this environment are very reactive and it's all about being fast. Diffusion is popular? Then let's do diffusion for [my specialty], or transformers for [X]. Or combine LLM with [my last topic]. People are constantly pivoting and jumping from one opportunity to the next. In my experience gone are the days where you could carve out a niche and work in peace on it for 4-5 years and have a consistent overarching story in your PhD.
- lmeyerov 2y ago
  IMO that kind of thinking flips correlation/causation, and feels like some sort salience bias towards voluminous but crappy work. Yes, you can do it, just good bye impact. Worse, it's a self-fulfilling prophecy: by moving your work to be more incremental, you're even more at risk by the incremental advances of others. The field exploded, so yes, there are way more good and even more bad papers, so some researchers interpret that as the need to publish a lot. Top labs attract a lot of students and funding, so the PI spends little time per student, and instead farms out to post docs who are there for only a few years, and in turn they do most of the phd student mentoring. And many folks want to bean count neurips papers. The result is yes, if you're unintentional about impact, and get distracted by the volume, you too can optimize for that, do lets of incremental work, ultimately you've learned some skills and wrote papers that won't matter. Google/FB/OpenAI/Spotify/LinkedIn/etc will still hire you at the end, that's fine, they hire a ton of phds. Objectively, there are some real jumps happening, but zoomed out, very few of the papers coming out in a given year change all that much. E.g., we're using LLMs for some hard societal problems (misinfo, cyber, ...), but nothing fundamental changed in our startup's work since gpt-4 came out, and all versions before gpt-4 weren't significant enough to motivate using generative LLM methods. There are a few methods we're tracking (e.g., graph RAG, certain kinds of agents), but most work is embarrassingly shallow or bad, so the incremental papers have done almost nothing to influence us, and we learn more from other industry folks doing more ambitious or real work here. The previous disruption on our AI side was around GNNs, but that was over many years and quite rocky. So if we were doing all this as a serious PhD project (invention) instead of as a startup (innovation), I'd expect similarly low levels of disruption from the perceived fast pace of others.

QuadmasterXLII 2y ago

If no one did the research that anyone could do, it would never get done.

marcosdumay 2y ago
In most markets, it pays to be contrarian. ... and when everybody decides to be a contrarian, the contrarian position is to do the popular thing. But the catch is that this never happens on practice.
lmeyerov 2y ago
As I wrote, different folks will already be chasing more immediate projects as they're too attractive: Think masters students, industry labs, startups, etc. Rules are meant to be broken. For example, a professor will often put a masters student on a cute me-too one-off project in a hot area as the short-termism helps the student have a smoother time, and the professor is trading expected low impact in the the project for a cheap shot to see if there is a more interesting question behind the immediate one. Much of the PhD is about learning to spot & navigate problems, not solutions. Young PhD students can totally pick the same me-too problems of masters students. The advisors would likely just think the student is using the same approach to seeing where the PhD worthy problem is. If still nothing, should be ready to drop the topic, or if they truly love me-too work more than the PhD process, leave for a better environment for that, like a company / industry lab / startup throwing more resources at the problem. Also, not every phd needs to do the original topic of attempting impact. You can totally use the PhD years to learn how to do me-too & unimpactful work or even the most weird & niche problems: as long as at least one advisor thinks they can stay funded while supporting you, most programs will let you through.
bonoboTP 2y ago
You're right, but the individual has to ask if entering a crowded niche is worth it or not. Does your main strength lie in absurd workload to grind out the paper a few months/weeks earlier to Arxiv than the next group will? The risk reward calculus still often benefits this approach. The risk is that you get scooped. If you do something nonobvious, the risk is that it doesn't pan out or the reviewers don't get the point. The latter is harder to solve by grinding. So many phds will chose the former, as at least putting in ridiculous hours feels more actionable. Also, doing the straightforward thing and getting scooped is less ego-hurting than failing at your own special pet idea. I think many who aren't in grad school don't understand how extremely important it is to have publications. At the beginning you must get publications to do internships and to build contacts at conferences. Then later it becomes imperative in order to graduate and get your next position. Nowadays you must even have multiple top conference papers just to start a PhD. Of course from the point of view of the advisor with 15 PhD students, the ideal student would do high risk high reward work and if 3 projects become super successful, 5 become quite successful and 7 yield nothing, that's still great for the advisor. But the 7 who lost several years would rather not go through this.

bonoboTP 2y ago

I find that in AI / ML / computer vision, the thinking horizon has shrunk a lot. Reviewers prize straightforward incremental work that nevertheless beats the SOTA. New problem formulations are too hard to decipher and understand deeply and nobody has time nowadays. Everything is super sped up. There's a fire hose of papers coming out that all do slightly different things, or the same concurrently. In this area even the best, most respected professors don't have long term laid out research plans. Grant plans must exist but they never quite pan out that way. Researchers in this environment are very reactive and it's all about being fast. Diffusion is popular? Then let's do diffusion for [my specialty], or transformers for [X]. Or combine LLM with [my last topic]. People are constantly pivoting and jumping from one opportunity to the next. In my experience gone are the days where you could carve out a niche and work in peace on it for 4-5 years and have a consistent overarching story in your PhD.

lmeyerov 2y ago
IMO that kind of thinking flips correlation/causation, and feels like some sort salience bias towards voluminous but crappy work. Yes, you can do it, just good bye impact. Worse, it's a self-fulfilling prophecy: by moving your work to be more incremental, you're even more at risk by the incremental advances of others. The field exploded, so yes, there are way more good and even more bad papers, so some researchers interpret that as the need to publish a lot. Top labs attract a lot of students and funding, so the PI spends little time per student, and instead farms out to post docs who are there for only a few years, and in turn they do most of the phd student mentoring. And many folks want to bean count neurips papers. The result is yes, if you're unintentional about impact, and get distracted by the volume, you too can optimize for that, do lets of incremental work, ultimately you've learned some skills and wrote papers that won't matter. Google/FB/OpenAI/Spotify/LinkedIn/etc will still hire you at the end, that's fine, they hire a ton of phds. Objectively, there are some real jumps happening, but zoomed out, very few of the papers coming out in a given year change all that much. E.g., we're using LLMs for some hard societal problems (misinfo, cyber, ...), but nothing fundamental changed in our startup's work since gpt-4 came out, and all versions before gpt-4 weren't significant enough to motivate using generative LLM methods. There are a few methods we're tracking (e.g., graph RAG, certain kinds of agents), but most work is embarrassingly shallow or bad, so the incremental papers have done almost nothing to influence us, and we learn more from other industry folks doing more ambitious or real work here. The previous disruption on our AI side was around GNNs, but that was over many years and quite rocky. So if we were doing all this as a serious PhD project (invention) instead of as a startup (innovation), I'd expect similarly low levels of disruption from the perceived fast pace of others.