9 ms·
Choose your weapon: Survival strategies for depressed AI academics
- monero-xmr 3y agoI know a handful of computer science academics personally, and each of them has founded at least one startup. They can go on sabbatical easily, do the startup, then come back once they sell it or it fails. It's a pretty sweet safety net. If I was an AI professor, I would go get a $1 million comp package on sabbatical then come back. The bubble will likely burst in 1 to 3 years, so you need to get paid now.
- sanxiyn 3y agoWhy do you think "bubble" will burst?
- ricopags 3y agoWhy do you use quotes? My reason would be NVDA sticker price.
- lupire 3y agoWe've had 20+ years of industry vastly outpaying academia.
- squigz 3y agoThis is exactly what I was thinking - when is the "bubble" for IT in general going to burst?
- pixl97 3y agoWhen software stops eating the world. No one sees that date on the horizon yet.
- squigz 3y agoI don't foresee technology going anywhere, no
- hattmall 3y agoIt has reasonably already burst a few times. At this point IT is just a reactionary part of the broader economy. IT being a leading technology experiences bubbles with in itself at as new flavors come around just like any other industry. Like in 2017 there was a huge bubble in tile work with schluter all in one systems. Any reasonable tile layer could take two weekends of classes and start making about $1000 a day installing these new tile systems in commercial buildings. Now the hype is dead most tile jobs are just as they were. There was a bubble in insurance a few years ago with Medicaid advantage plans as well. Perhaps IT has more bubbles but on the whole comparatively salaries are falling in line with other industries.
- squigz 3y agoAre they? https://codesubmit.io/blog/the-evolution-of-developer-salaries/ https://codesubmit.io/blog/the-evolution-of-developer-salari...
- monero-xmr 3y agoI have witnessed the same excitement and "never ending summer" psychology many times before. The cold winter always comes. We even had a big bubble recently with AI itself regarding self driving cars. It's amazing how every tech luminary went on stage and declared "This Year For Sure" to be when self driving cars would pick us all up... in 2019. It's perpetually 5 years away from mass deployment. Training LLMs has huge capital expenditure (capex) and is not producing revenues. As an anecdote, my brother in law works for a large 15k person Microsoft software shop, and they got a discount on their license renewal (yes, a discount) for including Github Copilot for every software engineer. We surmised this is to goose the number of "paying" licenses to help metrics. Extremely high salaries - extremely high capex - extremely high hype and gushing press - extremely nonexistent revenues (except for the pick and shovel sellers like NVDA) - this is a quintessential bubble in the making!
- pixl97 3y agoEh, Until we have a "this is AGI" moment, self driving cars are going to be a pipe dream, so not sure if they are a great example. And, yea, put your money where you're mouth is with your investments... I think you'll quickly end up relearning what "The market can remain irrational far longer than you can remain solvent" actually means. Trying to put discounts on license renewals on LLMs not selling is missing what the hell is going on in the market outside of AI. It is rough out there and business spending in general is way down. Trying to lump that in with lack of AI adoption may be a mistake.
- nopinsight 3y ago“OpenAI's annual run rate—a measure of one month's revenue multiplied by 12—hit the $2 billion revenue mark in December 2023, two people familiar with the knowledge told the Financial Times.” https://qz.com/openai-will-make-2-billion-revenue-soon-tech-ai-chatgpt-1851247985 https://qz.com/openai-will-make-2-billion-revenue-soon-tech-... Several well-known professional services firms have already deployed AI for their workforces. Many software engineers, esp younger ones, use ChatGPT or its equivalent every day at work.
- benreesman 3y ago
- skywhopper 3y agoBecause they always do. Because if you're looking, you'll see all the weaknesses in the current generation of LLM tech. But even if everything were great about LLM tech and the promise was as unlimited as Sam Altman promises, the mere existence of a skyrocketing field attracts scammer founders and investors who drive valuations higher until they are unsustainable.
- callalex 3y agoWhen the CEO of Microsoft starts claiming insane things like “AI is a bigger invention than fire and electricity” it’s a huge red flag that investors are rabid and not based in reality.
- kjkjadksj 3y agoYou don’t even have to go on sabbatical to found and run multiple startups.
- monero-xmr 3y agoI am acquaintances with a professor who is on multiple company boards. But really he is on meaningless committees, like the "Compensation Committee", and they just use him to broadcast the fact that an accomplished professor from a world-renowned university is "on their board". He just gets a check for them using him for marketing!
- math_dandy 3y agoAnother popular option is to join the university administration. Become an Associate Dean. (They are legion.)
- lupire 3y agoThe premise is bizarre. Academia is a sanctuary for people who can't get paid to do research because it's not economically exploitable, but is considered socially valuable. Academics are deeply lucky to have this option. Most of us can't get subsidies for our passion work. There's no reason to be "depressed" about corporations throwing mountains of money at you to come out of the sanctuary.
- benreesman 3y agoAgreed…mostly. It’s not irrational that we’ve forgotten how to funnel an American core value of “being best” via government spending on “defense” into “socialism” (the public good) by way of smart people who insist on comfortable and esteemed as their minimum job perk level, but care more about knowledge than power deep down.
- mlboss 3y agoVery nice post. One can the same advice for a startup that is trying to compete with a large company.
- modeless 3y ago> Obviously there are limits to what a small model is capable of doing This is not obvious to me. I mean yes, limits exist, but I seriously doubt that we are anywhere close to them. This is exactly the kind of question that needs academic research! I feel that our current models waste tons of compute and memory and there ought to be room to optimize them by 10x or even much more with new techniques.
- jszymborski 3y agoBy leaning on inductive biases, we've historically been able to get pretty far with relatively small models and litte data. The trouble is, applying inductive biases requires that you make assumptions about the problem, which requires that you understand the problem. The stronger the inductive bias, the stronger the assumptions. Creating fantasticly large models allows us to eschew the inductive bias in favour of model capacity. That capacity would lead to useless, fragile models which overfit without mountains of data. It's great to explore the maxima of either approach so we can define some kind of loose Pareto frontier of model size and performance. I do think there's tonnes of room to benefit from the pissing contest the tech giants are currently engaged in and learn from the massive models what can be done for smaller models. The RAG pipeline is kind of an approach to add some inductive bias to such large models. Currently people tend to use the same giant causal LLMs for RAG as they do for non-RAG Chat purposes, but I'm certain you can get much smaller models performing very well for RAG. In fact, I'm sure plenty of folks are doing it already considering the number of eyeballs on LLM and RAG atm.
- wavemode 3y agoI have the same feeling. I often wonder about the difference between learning and reasoning, in neural networks. If we could train a small network on a small amount of code and it figures out how to write code that's as good as what GPT-4 can write, that's what I'd call reasoning (and that's what's currently a massive open research question). On the other hand, it's definitely a lot easier to just show a gigantic model a gigantic amount of code until it becomes familiar with most types of applications a user would likely ask it about. So it feels like right now we're papering over our lack of research success by (essentially) throwing money at the problem, and it seems to be working well enough. (At least, well enough to be useful to people.)
- jszymborski 3y agoI think this is incredible advice. I have and continue to take approach IV and VIII[0], and its worked very well for me. Especially now that my research is investigating LLMs[1], scaling down and using architectures that focus on performance is paying dividends. Working in a niche applications (molecular biology) is also a great way to remain sane (although it comes with its own problems). [0] "Scale down" and "Small models" [1] Well, "large" is a relative term here.
- deleted 3y ago[deleted]
- __lbracket__ 3y agoYay ! Let us be depressed that postdocs we could have exploited for $16k/year have better options than polishing our shoes :(
- jlglover 3y agoAs someone in an adjacent field who tried to keep up with the state of the art until the late 2010s, I really relate to this article. I'm reminded of another interesting personal story about AI-induced academic depression https://arxiv.org/abs/2003.01870 https://arxiv.org/abs/2003.01870. I think it's valuable to record things like these.
- armchairhacker 3y agoTo me, academia has always been about developing proof-of-concepts for techniques that industry adopts years later. These proof-of-concepts are intricate but small, so they require a lot of ingenuity but not much resources or man-hours (a company can also hire far more employees than a professor). AI is no exception. All evidence suggests that current LLMs don't scale (EDIT: they do scale, but at a certain point somewhere around GPT4 the scaling slows down very quickly), so we need new techniques to fix their (many) flaws. A proof-of-concept for a new technique doesn't need hundreds of millions of dollars to demonstrate huge potential: such a demonstration only needs a small model using the new technique, a small model using the current SOTA, and some evidence that it scales (like slightly larger models that show the new model isn't slowing down its scaling vs SOTA). Academia is also about creating better explanations for things we already have. So researchers don't even need to develop new models, they can simply create small existing models and demonstrate some new property or explanation to get a worthwhile result. We probably need better explanations for how and why current LLMs work in order to create the better models mentioned above. EDIT: At least how it's supposed to work, you don't even need to show success either. Academics merely need to show a new technique with the potential to improve the SOTA. Even if the technique is a huge failure, the work still advances the future of AI, by contributing to our understanding of models (what works and what doesn't) as well as removing one potential option.
- famouswaffles 3y ago>AI is no exception. All evidence suggests that current LLMs don't scale Lol that's the opposite of what the evidence shows.
- BenFranklin100 3y agoYou got it exactly right. Academics have the freedom to pursue avenues and interests that have no clear immediate application nor profit.
- pama 3y agoI have seen no evidence that LLMs don’t scale. What evidence did you have in mind? Do you mean something specific and perhaps different from what I would call scaling? It seems in the last two years people got better at squeezing out more performance from smaller (but still large) language models but the continuous scaling of large models to multi modal applications has been rather impressive.
- rNULLED 3y agogreat piece, cites some important work
- darkoob12 3y agoI am an AI researcher working for a small company. When ChatGPT came out and people started to solve all sort of problems with prompting, I questioned my role and reasoned that in future middle-level researches will lose their jobs and if I want to stay in this business I have to upgrade my skills and possibly start publishing papers and get a PhD. Since in future only big tech companies would do research in AI and many automation problems of small companies would be solved with foundation models. And the competition for research jobs at big companies would be very tight.
- MrSkyNet 3y agoWhats your job as a AI researcher for a small company? When you mentioned prompting, i imagined you using vertex ai or smiliar 'slightly' lower level AI tools, but thats more ML Ops for me than AI research or AI integrator.
- darkoob12 3y agoWe develop information extraction models and solve Document AI problems. But many NLP and Vision tasks can be done with foundation models by simple fine-tuning, for which you need to know very basic definitions do not need to have a post-graduate degree.
- lupire 3y ago"Prompting" created a huge new area of AI research. The field is getting huge, not shrinking. Your comment is like saying hardware engineering is useless now because the microchip was invented.
- darkoob12 3y agoI haven't found any scientific value in prompt engineering papers I read.
- resource_waste 3y agoLets not pretend this is the same skill set. Prompting is yet another layer of abstraction. Untrained teenagers obsessed with chatGPT likely rival OP in prompting. OP can likely write some random forest algorithm, its just not needed anymore. I'm not an AI person, just a programmer, and I've found my time has gone into learning parameters to fine-tune + learning prompting. Maybe the parameters would have helped if I knew AI, but these are all layers that need to be learned.
- MrSkyNet 3y agoThe solution: start doing this with others. Seti for AI Research or just invent a solution, you are researchers!
- bbor 3y agolol. Symbolic approaches don’t even appear… crazy how oblivious people can be, even experts. Great article tho, well written otherwise!
- julienchastang 3y agoThere was a related article [0] about this in the WSJ a while back. Here are a couple of relevant quotes from that article: "Despite this progress, we remain concerned that there is a disproportionate amount of interest by policy makers in the voices of industry leaders rather than those in academia and civil society." "Furthermore, to truly understand this technology, including its sometimes unpredictable emergent capabilities and behaviors, public-sector researchers urgently need to replicate and examine the under-the-hood architecture of these models. That’s why government research labs need to take a larger role in AI." [0] https://www.wsj.com/tech/ai/artificial-intelligence-united-states-future-76c0082e https://www.wsj.com/tech/ai/artificial-intelligence-united-s...