5 ms·

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance com

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along. These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. And people will build the next GPT or whatever and even that’ll get boring. The people in big tech have to pivot and do whatever the economy demands. Whereas the academics can go back to writing papers like nothing changed.

ChatGTP 4y ago
So endless upside might not actually exist ?
- jimsimmons 4y ago
  It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.
  - bartwe 4y ago
    That hasn't been true for optimizing compilers either.
    - ChatGTP 4y ago
      Sorry what hasn’t been true ?
  - visarga 4y ago
    I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones. The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
    - anothernewdude 4y ago
      And? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
      - revelio 4y ago
        You can specify logit biases via the API, but yes it's limited.
  - sgt101 4y ago
    10^16 -> 10^24 flops isn't 100x, it's 100,000,000x This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT). On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...
    - jimsimmons 4y ago
      A lot of it is also backlog: modern GPUs are optimised for these workloads in a way old models weren’t
bjornsing 4y ago
> These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. This was my thought too. It cost approximately $4.75 billion to build the Large Hadron Collider at CERN. My educational background is in physics, but I would not be the least bit upset if a big chunk of public research funding was shifted from physics to CS/ML/AI. It’s obviously far more important to the future of humanity than finding the Higgs boson.
- speed_spread 4y ago
  I don't know, core physics leading to Fusion reactors and FTL travel seen much more important to me than AI that can make convincing robocalls and generate furry porn stories.
  - p1esk 4y ago
    How about AI that can do physics better than any human?
    - bannedbybros 4y ago
      [dead]
numbers_guy 4y ago
I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).
- mr_gibbins 4y ago
  As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years. Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment. Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).
  - jksk61 4y ago
    lmao even in my university (the serbian uni), we have at least calculus+linear algebra before any nn course. also to "learn" what's a cnn you just need gradients not integrals (unless you use some kind of non-lipschitz function as activation?), plus the idea of what a convolution is... but even Mobius knew it back in the 800s. anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.
martopix 4y ago
This isn't whining. It's about understanding what their role should be in this context.

ChatGTP 4y ago

So endless upside might not actually exist ?

jimsimmons 4y ago
It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.
- bartwe 4y ago
  That hasn't been true for optimizing compilers either.
  - ChatGTP 4y ago
    Sorry what hasn’t been true ?
- visarga 4y ago
  I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones. The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
  - anothernewdude 4y ago
    And? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
    - revelio 4y ago
      You can specify logit biases via the API, but yes it's limited.
- sgt101 4y ago
  10^16 -> 10^24 flops isn't 100x, it's 100,000,000x This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT). On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...
  - jimsimmons 4y ago
    A lot of it is also backlog: modern GPUs are optimised for these workloads in a way old models weren’t

bjornsing 4y ago

> These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. This was my thought too. It cost approximately $4.75 billion to build the Large Hadron Collider at CERN. My educational background is in physics, but I would not be the least bit upset if a big chunk of public research funding was shifted from physics to CS/ML/AI. It’s obviously far more important to the future of humanity than finding the Higgs boson.

speed_spread 4y ago
I don't know, core physics leading to Fusion reactors and FTL travel seen much more important to me than AI that can make convincing robocalls and generate furry porn stories.
- p1esk 4y ago
  How about AI that can do physics better than any human?
  - bannedbybros 4y ago
    [dead]

numbers_guy 4y ago

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).

mr_gibbins 4y ago
As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years. Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment. Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).
- jksk61 4y ago
  lmao even in my university (the serbian uni), we have at least calculus+linear algebra before any nn course. also to "learn" what's a cnn you just need gradients not integrals (unless you use some kind of non-lipschitz function as activation?), plus the idea of what a convolution is... but even Mobius knew it back in the 800s. anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.

martopix 4y ago

This isn't whining. It's about understanding what their role should be in this context.