12 ms·
Choose Your Weapon: Survival Strategies for Depressed AI Academics
- roschdal 4y ago[flagged]
- cloudhan 4y agoThis papar is really interesting, especially so when you scroll to the Reference part ;)
- etiam 4y agoRight. Also, note the publication date...
- nicce 4y agoNever Gonna Give You Up. Never gonna let you down...
- KineticLensman 4y agoUnfortunately, having checked Ref 7, I now can't unsee Chuck Tingle.
- jimsimmons 4y agoAcademics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along. These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. And people will build the next GPT or whatever and even that’ll get boring. The people in big tech have to pivot and do whatever the economy demands. Whereas the academics can go back to writing papers like nothing changed.
- ChatGTP 4y agoSo endless upside might not actually exist ?
- jimsimmons 4y agoIt may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.
- bartwe 4y agoThat hasn't been true for optimizing compilers either.
- ChatGTP 4y agoSorry what hasn’t been true ?
- visarga 4y agoI thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones. The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
- anothernewdude 4y agoAnd? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
- bjornsing 4y ago> These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. This was my thought too. It cost approximately $4.75 billion to build the Large Hadron Collider at CERN. My educational background is in physics, but I would not be the least bit upset if a big chunk of public research funding was shifted from physics to CS/ML/AI. It’s obviously far more important to the future of humanity than finding the Higgs boson.
- speed_spread 4y agoI don't know, core physics leading to Fusion reactors and FTL travel seen much more important to me than AI that can make convincing robocalls and generate furry porn stories.
- p1esk 4y agoHow about AI that can do physics better than any human?
- bannedbybros 4y ago[dead]
- numbers_guy 4y agoI don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).
- mr_gibbins 4y agoAs someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years. Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment. Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).
- jksk61 4y agolmao even in my university (the serbian uni), we have at least calculus+linear algebra before any nn course. also to "learn" what's a cnn you just need gradients not integrals (unless you use some kind of non-lipschitz function as activation?), plus the idea of what a convolution is... but even Mobius knew it back in the 800s. anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.
- martopix 4y agoThis isn't whining. It's about understanding what their role should be in this context.
- vintermann 4y agoThis paper is 1. Really a blog post disguised as a paper. 2. Silly. 3. Excellent.
- sanxiyn 4y agoI expected this to include more resources like TPU Research Cloud, but it doesn't mention even TRC itself. Disappointing. https://sites.research.google/trc/about/ https://sites.research.google/trc/about/
- blackbear_ 4y ago> the grant funding structure is such that it rewards safe and incremental research on popular topics [...] Therefore, universities should probably avoid making grant funding a condition for hires and promotions There you go. Just fix this idiotic "publish (NeurIPS) or perish" attitude already!
- MrScruff 4y agoObviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. Of course, any rational person might have expected this given how the one working example came into existence. In general it all looks like a pretty resounding endorsement of the David Chalmers position to me.
- hakuseki 4y ago> Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network. To get intelligent behavior, it's not sufficient to wire together a large neural network. You also need to use an optimizer to train it on a large data set.
- llamataboot 4y agoonce upon a time the optimizer was die or reproduce...now it's a little faster and more convoluted...
- MrScruff 4y agoYes of course, I'm being loose in my language, but you know what I'm getting at.
- galaxyLogic 4y agoI think "Dial F For Frankenstein" makes your point clear. It is absurd to think the telephone network would just emergently become conscious when there are enough phones it connects together. https://www.historyofinformation.com/detail.php?id=2137 https://www.historyofinformation.com/detail.php?id=2137
- melbourne_mat 4y agoI can imagine AI academics are in a tough spot. However what about the existential angst the rest of us - who don't even do anything AI related on a day to day basis - are feeling? I think big changes are coming and it's not gonna be pretty.
- lewhoo 4y agoPapers will be written on how different reactions to AI emerged in all this noise. Unfortunately, I'm all in (unwillingly) for the doomers perspective. I really can't help it but I'd love to be proven wrong. Altman keeps saying intelligence will be cheap and all I hear is I'm heading to (one day, some day) become the horse and carriage of cognitive processes.
- wing-_-nuts 4y agoMy big question is what is your standard java / c# dev supposed to do to keep up with the massive changes that are coming? I was seriously considering doing a GT OMCS, but I don't know what sort of return on investment that would have for me. I was also considering building a gaming pc to also be able to play around with ai / ml, only, I'm not even sure what one would be able to run locally anymore even with a 4090? Overall? I feel you. It makes me worried for the new grads today. I don't really know if this old dog is up for new tricks.
- ericmcer 4y agoCan't they just jump into one of the thousands of new high paying jobs that have been created for their specific niche in the last few months? I am having trouble being sympathetic to AI researchers that AI is exploding.
- fancyfredbot 4y agoI think this paper is a joke but like all good jokes has a grain of truth. I would imagine that most AI academics are very excited by recent developments, if you worked on conversational language models you are more likely to view recent progress at OpenAI as a vindication than a threat. That said there must be some sadness that it wasn't you.
- layer8 4y agoI think the frustration is less about that they weren’t involved in the recent successes, but rather that they can’t build on models like GPT due to their scale and non-openness.
- jcfrei 4y agoBetter give up now as a scientist so you got an early lead into becoming a chef. Or one of the other jobs where a human touch is still valued.
- dougb5 4y agoI love that the citation to GPT-4 is a Rickroll :)
- est31 4y agoI think it's always been the case that if industry really gets interested in making progress in some particular area, they make it, and way faster than academia ever could. Not just in computer science but also in fields like chemistry or engineering. Academia is great for doing basic science where there is no profit motive for industry to do something about it. That's what the niche/"few care about" sections talk about, where IMO the strength of academia lies. computer science is I think in general comparatively in a good position compared to other sciences in how easy it is to get funded.
- nullsense 4y agoI like how they release what reads more like a blog post as a paper.
- bo1024 4y agoAs a theorist in AI, the premise is funny but also really sad. And yet it is a very pervasive premise in the field. OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process. It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh'd and aah'd and threw up their hands: "we don't have the resources to build a bigger rocket; how can we do science?" Unfortunately, the research field of AI is diseased: it places way too much value on showing off big shiny things above progress in scientific understanding. But there is a ton of progress available to be made by small teams on small budgets. There is so much we don't understand about neural networks alone, even small ones.
- gitfan86 4y agoOpenAI wasn't created to "Do Science". It was created to give humanity a better chance at navigating the singularity.
- deleted 4y ago[deleted]
- agravier 4y agoOoooh and here I was, an ignoramus thinking OpenAI was mostly trying to find a way to milk the generative models cash cow. Silly, cynical me, I should have known better.
- muskmusk 4y ago> If we have learned one thing from deep learning, it is that scaling works. From the ImageNet [15] competitions and their various winners to ChatGPT, Gato [13], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results. (By the time you are reading this, that list of very recent AI milestones might very well be outdated.). Of course there are improvements to learning algorithms and network architectures as well, but these improvements are only really useful in the context of the massive scale of experiments. (Sutton talks about the “Bitter Pill”, referring to that simple methods that scale well always win the day when more compute becomes available [18].) A scale that is not achievable by academic researchers nowadays. As far as we can tell, the gap between the amount of compute available to ordinary researchers and the amount available to stay competitive is growing every year. >This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing. This sounds like teenagers whining they can't all be popular. Science isn't a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn't you discovers something useful that isn't bad because then you can't discover it. It is good because that is a problem solved. The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware/parameters/whatever. The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!
- pixl97 4y ago> Science isn't a competition to be won so that you can get praise and attention. Conversely business seems to be a competition which wants exactly that, no matter what the long term consequences are. Lead in gasoline makes the engine stop knocking, great. Problem solved, nothing to worry about, right?Don't worry about funding scientists to look into it, it's all going to be ok.
- chubot 4y agoThat's the first time I've been rick roll'd reading an academic paper :) (citation 1 for GPT-4)
- mensetmanusman 4y agoAfter a PhD@MIT and then entering industry, it has been my general understanding that industry is ahead on any scientific fronts that are profitable in fields that have a progress<>profit feedback loop. Academia still plays an important training ground role, and it could shift focus to those areas that could be profitable but require coordination that is generally not feasible among certain institutions.
- mensetmanusman 4y agoAI profs should start working with the bio folks now and making thinking materials that are orders of magnitude more power efficient than silicon.
- rafaelero 4y agoDon't worry. Soon governments will start pouring money into this field. They still won't be competitive, but at least there will be enough computing power to play with.
- tpoacher 4y agoGotta love a paper which cites Rick Astley's "Never Gonna Give you Up". Properly, for that matter.
- thunkshift1 4y agoThus is medium shit post at best.. why is it on arxiv
- ralphc 4y agoAI fatigue is the new JavaScript fatigue.
- nothrowaways 4y agoThe only limitation of this scientific article is the fact that it is not two-columns format.