6 ms·

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a hi

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).

mr_gibbins 3y ago
As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years. Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment. Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).
- jksk61 3y ago
  lmao even in my university (the serbian uni), we have at least calculus+linear algebra before any nn course. also to "learn" what's a cnn you just need gradients not integrals (unless you use some kind of non-lipschitz function as activation?), plus the idea of what a convolution is... but even Mobius knew it back in the 800s. anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.
  - dontupvoteme 3y ago
    >just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon. Sounds dead on. Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data? I don't fault Chomsky at all for being fed up with the hype here. The entire field is also glossing over the fact that other languages which aren't English exist.
    - TeMPOraL 3y ago
      >> anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. GP here is, IMO, confusing what the corporations want (1), with what corporate R&D people want (2). As long as the corps see good ROI on throwing infinite money at their AI R&D departments, then those corporate researchers are better positioned and better equipped to do actual, solid science, than academia ever can be. This has happened many times before, including in this industry. Research is best done by well-funded teams of smart people left to do whatever they fancy. When those conditions arise, progress happens, and it doesn't matter whether it's the government or industry that creates them. (Conversely, the best hope for academia to become relevant again is that corporations lose interest in this research, and defund their departments. This could happen if e.g. transformers end up being a dead end, or compute suddenly becomes very expensive.) > Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data? The latter. And guess what, they're not trying to solve the issue of linguistics. They started as tools to generate human-sounding text, but in the process of just throwing more data and compute at them, they not only got better, but started to acquire something resembling concept-level understanding. It turns out that surprisingly many aspects of thinking seem to reduce well to proximity search in a vector space, if that space is high-dimensional enough. This result is both surprising and impactful well beyond the field of AI. It's arguably the first potential path we identified that the evolution could take to gradually random-walk itself from amoeabas to human brains.
- sarchertech 3y ago
  > Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. In the US I’ve never seen a BS in computer science that didn’t require calculus. I can’t speak for the UK, but it would surprise me that what you say is true.
- johnaspden 3y ago
  > To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Oh the poor dears, imagine needing schoolboy maths to do science.
Tomis02 3y ago
> Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway That's interesting, could you please elaborate?
- numbers_guy 3y ago
  It's not interesting. You can regard OpenAI as one of the most prestigious AI labs in the world and only the best of the best get a chance to work there.
  - aiisjustanif 3y ago
    It’s very interesting that you think most people with a PhD working in AI / ML / Data Science academically can’t get a job at OpenAI. You can easily see a lot of people who work at OpenAI on LinkedIn. University AI labs are always left out because non-commercial products just aren’t as noticeable. I highly doubt that many of these academics would struggle being amount candidates at DeepMind, OpenAI and FAIR. Prestige is very subjective, which makes it also very broad thankfully.
matthewdgreen 3y ago
I think your response is very narrow in defining what a "really cool problem" is. If in the 1970s you imagined that the cool problem in ECE/CS was "building ever-more-powerful computer processors" then you basically got smoked by Intel and other industry labs. But there's a different way to look at it. Because industry went off and solved the now-"boring" problem of building ever-more-powerful computer processors, the people left behind in academia got to invent machine learning, public-key cryptography, modern coding theory, distributed systems... and so on. What makes academic research valuable is not the freedom of "I get to plan my own day", but the freedom of "I get to work on the weird problems that industry doesn't even realize are problems."
- oldsecondhand 3y ago
  Public key cryptography was invented in the 70s, commercialized in the 80s. In machine learning we already had back-propagation in the 1970, but had to wait another 40 years for Intel and Nvidia to create ever more powerful processors to tackle useful problems.
- numbers_guy 3y ago
  OK, I agree with you. But this sort of struggle that you highlight is common in many aspects of research. As a researcher you are conflicted between your curiosity instinct to explore new paths, and your desire to also have big results. Ideally you would have a balance between the two, although you can definitely make a career out of only one.
- TeMPOraL 3y ago
  The "cool problem" is cracking the mystery of intelligence, of consciousness. Building a general AI. Until recently, this was mostly an academic pursuit, and people expected it to take decades or more. But now, suddenly, it's highly likely the problem will be cracked within a decade, and it will be done by corporate R&D teams, by means of scaling up the transformer architectures. So I get how they feel - one of the coolest problems ever instantly went from being something anyone could hope to contribute to, to something almost no one can. You can't match the compute to do the things OpenAI & Google & friends do. And if you happen to stumble on something related that can be explored without access to obscene amounts of capital, guess what, the corporate research teams will notice it, and they can do it better than you, and then they can the idea much further than you ever could.

mr_gibbins 3y ago

As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years. Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment. Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).

jksk61 3y ago
lmao even in my university (the serbian uni), we have at least calculus+linear algebra before any nn course. also to "learn" what's a cnn you just need gradients not integrals (unless you use some kind of non-lipschitz function as activation?), plus the idea of what a convolution is... but even Mobius knew it back in the 800s. anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.
- dontupvoteme 3y ago
  >just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon. Sounds dead on. Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data? I don't fault Chomsky at all for being fed up with the hype here. The entire field is also glossing over the fact that other languages which aren't English exist.
  - TeMPOraL 3y ago
    >> anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth. GP here is, IMO, confusing what the corporations want (1), with what corporate R&D people want (2). As long as the corps see good ROI on throwing infinite money at their AI R&D departments, then those corporate researchers are better positioned and better equipped to do actual, solid science, than academia ever can be. This has happened many times before, including in this industry. Research is best done by well-funded teams of smart people left to do whatever they fancy. When those conditions arise, progress happens, and it doesn't matter whether it's the government or industry that creates them. (Conversely, the best hope for academia to become relevant again is that corporations lose interest in this research, and defund their departments. This could happen if e.g. transformers end up being a dead end, or compute suddenly becomes very expensive.) > Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data? The latter. And guess what, they're not trying to solve the issue of linguistics. They started as tools to generate human-sounding text, but in the process of just throwing more data and compute at them, they not only got better, but started to acquire something resembling concept-level understanding. It turns out that surprisingly many aspects of thinking seem to reduce well to proximity search in a vector space, if that space is high-dimensional enough. This result is both surprising and impactful well beyond the field of AI. It's arguably the first potential path we identified that the evolution could take to gradually random-walk itself from amoeabas to human brains.
sarchertech 3y ago
> Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'. In the US I’ve never seen a BS in computer science that didn’t require calculus. I can’t speak for the UK, but it would surprise me that what you say is true.
johnaspden 3y ago
> To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Oh the poor dears, imagine needing schoolboy maths to do science.

Tomis02 3y ago

> Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway That's interesting, could you please elaborate?

numbers_guy 3y ago
It's not interesting. You can regard OpenAI as one of the most prestigious AI labs in the world and only the best of the best get a chance to work there.
- aiisjustanif 3y ago
  It’s very interesting that you think most people with a PhD working in AI / ML / Data Science academically can’t get a job at OpenAI. You can easily see a lot of people who work at OpenAI on LinkedIn. University AI labs are always left out because non-commercial products just aren’t as noticeable. I highly doubt that many of these academics would struggle being amount candidates at DeepMind, OpenAI and FAIR. Prestige is very subjective, which makes it also very broad thankfully.

matthewdgreen 3y ago

I think your response is very narrow in defining what a "really cool problem" is. If in the 1970s you imagined that the cool problem in ECE/CS was "building ever-more-powerful computer processors" then you basically got smoked by Intel and other industry labs. But there's a different way to look at it. Because industry went off and solved the now-"boring" problem of building ever-more-powerful computer processors, the people left behind in academia got to invent machine learning, public-key cryptography, modern coding theory, distributed systems... and so on. What makes academic research valuable is not the freedom of "I get to plan my own day", but the freedom of "I get to work on the weird problems that industry doesn't even realize are problems."

oldsecondhand 3y ago
Public key cryptography was invented in the 70s, commercialized in the 80s. In machine learning we already had back-propagation in the 1970, but had to wait another 40 years for Intel and Nvidia to create ever more powerful processors to tackle useful problems.
numbers_guy 3y ago
OK, I agree with you. But this sort of struggle that you highlight is common in many aspects of research. As a researcher you are conflicted between your curiosity instinct to explore new paths, and your desire to also have big results. Ideally you would have a balance between the two, although you can definitely make a career out of only one.
TeMPOraL 3y ago
The "cool problem" is cracking the mystery of intelligence, of consciousness. Building a general AI. Until recently, this was mostly an academic pursuit, and people expected it to take decades or more. But now, suddenly, it's highly likely the problem will be cracked within a decade, and it will be done by corporate R&D teams, by means of scaling up the transformer architectures. So I get how they feel - one of the coolest problems ever instantly went from being something anyone could hope to contribute to, to something almost no one can. You can't match the compute to do the things OpenAI & Google & friends do. And if you happen to stumble on something related that can be explored without access to obscene amounts of capital, guess what, the corporate research teams will notice it, and they can do it better than you, and then they can the idea much further than you ever could.