5 ms·

I am really surprised by how tunnel visioned top scientists, VCs and CEOs are in the AI space. This whole movement of simply increasing compute or throwing in

by max_ 2y ago

I am really surprised by how tunnel visioned top scientists, VCs and CEOs are in the AI space. This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. This is like saying if human beings just flap their wings hard enough, they will be able to fly like birds. We need to finance original research, alternative research and think of working on an actual theory of intelligence. The stage AI is at right now is more like the Ptolemaic model of the universe. We need a Galileo and Kepler of AI. Maybe they already exist but are being marginalized by the current establishment?

juliangamble 2y ago
Perhaps the more apt analogy is that we need "a John Von Neuman" to come and organise the pieces to give us that leap.
- coffeebeqn 2y ago
  Either way - if/when that breakthrough happens would you rather have n compute or n^4 compute ready to go?
  - Retric 2y ago
    Buying compute before such breakthroughs is a terrible investment compared to just waiting.
    - imtringued 2y ago
      Just rent it out. It's not like we have an abundance of GPUs as of now.
    - coffeebeqn 2y ago
      It’s not like it’s sitting idle - the companies feel like they’re getting a ROI in whatever Gen AI nonsense they’re pushing to their apps
      - Retric 2y ago
        Same as it ever was. SV runs 80/20 on investors getting fleeced vs. vaguely realistic business models.
- silverquiet 2y ago
  Given that population has increased and arguably(?) gotten more intelligent since Neuman's day (and for all the deserved respect he gets, I'm more of a Feynman fan myself so there were a few brainiacs back then), you'd have to assume there are a few people on his level walking around today, and I wonder what they're up to. Is creating a more addictive website for a FAANG company or a better high-speed trading algorithm for a Fintech what they see as a good calling? The smartest person I knew in high school (and I went to a rather high end one) who easily was the valedictorian, basically only wanted to play the trumpet (which he was also superlatively good at). I don't really know what became of him.
  - A_D_E_P_T 2y ago
    > Given that population has increased and arguably(?) gotten more intelligent since Neuman's day, you'd have to assume there are a few people on his level walking around today, and I wonder what they're up to. Hahaha, oh man. Von Neumann, and many of his contemporaries, weren't secretive researchers toiling in obscure corporate labs -- they were famous public intellectuals with renowned accomplishments across many fields. There are no men of that caliber alive today, and the educational system absolutely excels in ensuring that such well-rounded men are not produced.
vendiddy 2y ago
I am an outsider to the space, but I feel this sentiment. It appears that a lot of breakthroughs came from further understanding how the brain works but we are just trying to brute-force our way to do harder things. (And our brains also do things that AIs cannot even when running on a few salads a day.)
sk11001 2y ago
> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. There's no way for you or for anyone else to know that it's a dead end. Also, it's not "simply throwing more compute" - the algorithms have evolved too and they'll continue to. Also also, it's not "hopefully attain AI results" - we have AI results now and they keep getting better. People are spending resources on a trajectory which has worked well - we've been cranking up the compute and better results keep coming out, why stop now?
- dpflan 2y ago
  One must consider the concept of "diminishing returns" eventually. Has Generative AI Already Peaked? - Computerphile (https://www.youtube.com/watch?v=dDUC-LqVrPU https://www.youtube.com/watch?v=dDUC-LqVrPU)
  - soared 2y ago
    Moores law is the inverse
    - Retric 2y ago
      Moores law doesn’t help when you’re spending money today rather than 10 years from now.
- max_ 2y ago
  All I am saying is that there are other strategies to explore. It's ok to have deep learning. But the herding towards deep learning is just too much. Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions" Scientific progress is made by researchers exploring other alternatives. Herding can only slow down progress.
  - jerpint 2y ago
    The imagenet results’ “main” contribution was showing that scaling the compute on large datasets worked surprisingly well. They didn’t contribute any new models as much as they showed that scaling is what helped these models learn and obtain SOTA. It’s no surprise that industry has been pushing for more compute ever since.
PoignardAzur 2y ago
I mean, this sounds really wise and all, but it would have sounded equally wise 3 years ago and yet someone making predictions based on this reasoning would have utterly failed to predict Dall-E, ChatGPT or Whisper.
Maro 2y ago
AI research in 2024, both fundamental and applied, is probably the most funded research in the history of mankind..
- max_ 2y ago
  Grant funders only finance projects with high probability of success. It's very difficult to get funding on projects that don't already have a track record of success. Which is why all research facilities working in AI today that I have read about only do research on Neural Networks
  - itchyjunk 2y ago
    Do you have any citation on lack of funding for foundational research? From the looks of things, it seems like there is insane amount of money in AI and while maybe majority of it goes to "projects with high probability of success", whatever that means, fringe ideas seem to generate good amount of capital as well. Even (what I would consider to be) academic projects seems to get good funding [0]. On top of that, it seems like you're not saying that Transformers might be a dead end but Neural Networks itself might be one. At this point, NNs already gives a lot of value. If someone's goal is to just get products out that say diffusion models already helps you achieve, how is that even a dead end? Do you mean dead end for AGI? In which case, there seems to be research on alternatives there as well? [1] [0] https://www.youtube.com/watch?v=rie-9AEhYdY https://www.youtube.com/watch?v=rie-9AEhYdY [1] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa
    - max_ 2y ago
      I don't have any citations. Just look at what is being published in Arxiv. Call your friends in university departments. Ask people working in startups or corporation. AI at this point is just Neural Networks. Do I think it's a dead end for AGI? I really don't know what AGI is, but for what ever new possibilities we can unlock. We need newer techniques that are less intensive computationally.
naveen99 2y ago
transformers don’t show any sign of convergence yet. The more data and compute we throw at them, weather it be chess, go, image classification, text prediction, the better they get. Tunnel vision is simply focus on finding a horizon, or a limit of some kind.
- jiggawatts 2y ago
  How can you be so confident when nobody has managed to convincingly beat GPT-4 yet, including Open AI themselves? All evidence is the throwing even textbook qualify data at a model of almost any size just approaches an asymptote just a tiny bit above GPT 4. A better model of some data starts to look increasingly like then data, not like something else beyond the data.
  - naveen99 2y ago
    i don't know about the future. I am only saying, we haven't seen diminishing returns from training transformers on more data / compute yet.
laeri 2y ago
If it is needed soneone will and probably is currently doing original research. The payoff is too great that alternatives would be ignored. As long as financial incentives exist progress will inevitably happen and the financial incentives in this case is massive.
truculent 2y ago
> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. Hopefully? The last 10 years of astonishing AI results have largely been driven by “throwing in more compute and data”.
Avshalom 2y ago
If it takes 100 billion to train and run an llm then only companies with 100 billion can train and run an llm. That's their moat.
cardanome 2y ago
I used to have the same opinion for many years. Then generative AI became good. Good enough for me to use it as a daily assistant. So good it scared me and led me to rethink my position. I have missed one crucial thing: Big enough changes in quantity do lead new qualities as well. Training models with insane amounts of data was the necessary step to finally make the qualitative leap into developing models that were practical for mainstream use. One of the big things holding back AI research has always been trying to be too clever. This article sums it up perfectly: > The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- max_ 2y ago
  Thanks. This blog is very interesting
slibhb 2y ago
> We need to finance original research, alternative research and think of working on an actual theory of intelligence. > We need a Galileo and Kepler of AI. Maybe they already exist but are being marginalized by the current establishment? The problem with "theories of intelligence" is that they need to yield results. You can come up with whatever brilliant theory you like but at the end of the day you need to produce a tool, a model, a result, something that does something. And that's exactly what "theories of intelligence" have failed to do. Your argument is like Chomsky whining in the NYT about machine learning. What did Chomsky's linguistics actually do? What did they enable us to build? The answer is probably not "nothing" -- it's more like "not much". Meanwhile, machine learning gave us a tool that can translate from one language to another reasonably well and didn't stop there. Theory isn't useless but we have plenty of it. There's so much borderline useless theory about language and intelligence in academia and it has produced so little tangible value. We absolutely do not need more of it.
deleted 2y ago
[deleted]
nebula8804 2y ago
Sam Altman has in the past talked about gatekeeping AI compute to focus on important problems: things like solving for vaccines or intense science or even just things that will fundamentally improve humans day to day vs asking it dumb questions that could be found with a google search.
Havoc 2y ago
> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. It’s the logical incremental step until an actual breakthrough is model on technique
GistNoesis 2y ago
>We need a Galileo and Kepler of AI. Let's imagine a modern day Galileo discover a new revolutionary algorithm that gives the same results for 1000x less compute. How would he transform this competitive advantage into some money without revealing the secret sauce ? Today's he will be in competition with people spending $10^8 to train. Even with the 10^3 cost reduction it's out of range for most people. And then he has to build a business around while keeping making sure his secret techniques don't leak. VCs and CEOs are just playing the controlling the business game, raising barrier to make sure they stay in control. Incremental advancements are great for them, because they get to stay in the lead while reducing the possibility that a revolutionary advancement comes to shuffle their game. Renaissance Technologies could maintain an edge over the whole stock market for plenty of years. AI-controlled company will probably be able too, but you won't hear about them.

by max_ 2y ago

juliangamble 2y ago

Perhaps the more apt analogy is that we need "a John Von Neuman" to come and organise the pieces to give us that leap.

coffeebeqn 2y ago
Either way - if/when that breakthrough happens would you rather have n compute or n^4 compute ready to go?
- Retric 2y ago
  Buying compute before such breakthroughs is a terrible investment compared to just waiting.
  - imtringued 2y ago
    Just rent it out. It's not like we have an abundance of GPUs as of now.
  - coffeebeqn 2y ago
    It’s not like it’s sitting idle - the companies feel like they’re getting a ROI in whatever Gen AI nonsense they’re pushing to their apps
    - Retric 2y ago
      Same as it ever was. SV runs 80/20 on investors getting fleeced vs. vaguely realistic business models.
silverquiet 2y ago
Given that population has increased and arguably(?) gotten more intelligent since Neuman's day (and for all the deserved respect he gets, I'm more of a Feynman fan myself so there were a few brainiacs back then), you'd have to assume there are a few people on his level walking around today, and I wonder what they're up to. Is creating a more addictive website for a FAANG company or a better high-speed trading algorithm for a Fintech what they see as a good calling? The smartest person I knew in high school (and I went to a rather high end one) who easily was the valedictorian, basically only wanted to play the trumpet (which he was also superlatively good at). I don't really know what became of him.
- A_D_E_P_T 2y ago
  > Given that population has increased and arguably(?) gotten more intelligent since Neuman's day, you'd have to assume there are a few people on his level walking around today, and I wonder what they're up to. Hahaha, oh man. Von Neumann, and many of his contemporaries, weren't secretive researchers toiling in obscure corporate labs -- they were famous public intellectuals with renowned accomplishments across many fields. There are no men of that caliber alive today, and the educational system absolutely excels in ensuring that such well-rounded men are not produced.

vendiddy 2y ago

I am an outsider to the space, but I feel this sentiment. It appears that a lot of breakthroughs came from further understanding how the brain works but we are just trying to brute-force our way to do harder things. (And our brains also do things that AIs cannot even when running on a few salads a day.)

sk11001 2y ago

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. There's no way for you or for anyone else to know that it's a dead end. Also, it's not "simply throwing more compute" - the algorithms have evolved too and they'll continue to. Also also, it's not "hopefully attain AI results" - we have AI results now and they keep getting better. People are spending resources on a trajectory which has worked well - we've been cranking up the compute and better results keep coming out, why stop now?

dpflan 2y ago
One must consider the concept of "diminishing returns" eventually. Has Generative AI Already Peaked? - Computerphile (https://www.youtube.com/watch?v=dDUC-LqVrPU https://www.youtube.com/watch?v=dDUC-LqVrPU)
- soared 2y ago
  Moores law is the inverse
  - Retric 2y ago
    Moores law doesn’t help when you’re spending money today rather than 10 years from now.
max_ 2y ago
All I am saying is that there are other strategies to explore. It's ok to have deep learning. But the herding towards deep learning is just too much. Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions" Scientific progress is made by researchers exploring other alternatives. Herding can only slow down progress.
- jerpint 2y ago
  The imagenet results’ “main” contribution was showing that scaling the compute on large datasets worked surprisingly well. They didn’t contribute any new models as much as they showed that scaling is what helped these models learn and obtain SOTA. It’s no surprise that industry has been pushing for more compute ever since.

PoignardAzur 2y ago

I mean, this sounds really wise and all, but it would have sounded equally wise 3 years ago and yet someone making predictions based on this reasoning would have utterly failed to predict Dall-E, ChatGPT or Whisper.

Maro 2y ago

AI research in 2024, both fundamental and applied, is probably the most funded research in the history of mankind..

max_ 2y ago
Grant funders only finance projects with high probability of success. It's very difficult to get funding on projects that don't already have a track record of success. Which is why all research facilities working in AI today that I have read about only do research on Neural Networks
- itchyjunk 2y ago
  Do you have any citation on lack of funding for foundational research? From the looks of things, it seems like there is insane amount of money in AI and while maybe majority of it goes to "projects with high probability of success", whatever that means, fringe ideas seem to generate good amount of capital as well. Even (what I would consider to be) academic projects seems to get good funding [0]. On top of that, it seems like you're not saying that Transformers might be a dead end but Neural Networks itself might be one. At this point, NNs already gives a lot of value. If someone's goal is to just get products out that say diffusion models already helps you achieve, how is that even a dead end? Do you mean dead end for AGI? In which case, there seems to be research on alternatives there as well? [1] [0] https://www.youtube.com/watch?v=rie-9AEhYdY https://www.youtube.com/watch?v=rie-9AEhYdY [1] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa
  - max_ 2y ago
    I don't have any citations. Just look at what is being published in Arxiv. Call your friends in university departments. Ask people working in startups or corporation. AI at this point is just Neural Networks. Do I think it's a dead end for AGI? I really don't know what AGI is, but for what ever new possibilities we can unlock. We need newer techniques that are less intensive computationally.

naveen99 2y ago

transformers don’t show any sign of convergence yet. The more data and compute we throw at them, weather it be chess, go, image classification, text prediction, the better they get. Tunnel vision is simply focus on finding a horizon, or a limit of some kind.

jiggawatts 2y ago
How can you be so confident when nobody has managed to convincingly beat GPT-4 yet, including Open AI themselves? All evidence is the throwing even textbook qualify data at a model of almost any size just approaches an asymptote just a tiny bit above GPT 4. A better model of some data starts to look increasingly like then data, not like something else beyond the data.
- naveen99 2y ago
  i don't know about the future. I am only saying, we haven't seen diminishing returns from training transformers on more data / compute yet.

laeri 2y ago

If it is needed soneone will and probably is currently doing original research. The payoff is too great that alternatives would be ignored. As long as financial incentives exist progress will inevitably happen and the financial incentives in this case is massive.

truculent 2y ago

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. Hopefully? The last 10 years of astonishing AI results have largely been driven by “throwing in more compute and data”.

Avshalom 2y ago

If it takes 100 billion to train and run an llm then only companies with 100 billion can train and run an llm. That's their moat.

cardanome 2y ago

I used to have the same opinion for many years. Then generative AI became good. Good enough for me to use it as a daily assistant. So good it scared me and led me to rethink my position. I have missed one crucial thing: Big enough changes in quantity do lead new qualities as well. Training models with insane amounts of data was the necessary step to finally make the qualitative leap into developing models that were practical for mainstream use. One of the big things holding back AI research has always been trying to be too clever. This article sums it up perfectly: > The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

max_ 2y ago
Thanks. This blog is very interesting

slibhb 2y ago

> We need to finance original research, alternative research and think of working on an actual theory of intelligence. > We need a Galileo and Kepler of AI. Maybe they already exist but are being marginalized by the current establishment? The problem with "theories of intelligence" is that they need to yield results. You can come up with whatever brilliant theory you like but at the end of the day you need to produce a tool, a model, a result, something that does something. And that's exactly what "theories of intelligence" have failed to do. Your argument is like Chomsky whining in the NYT about machine learning. What did Chomsky's linguistics actually do? What did they enable us to build? The answer is probably not "nothing" -- it's more like "not much". Meanwhile, machine learning gave us a tool that can translate from one language to another reasonably well and didn't stop there. Theory isn't useless but we have plenty of it. There's so much borderline useless theory about language and intelligence in academia and it has produced so little tangible value. We absolutely do not need more of it.

deleted 2y ago

[deleted]

nebula8804 2y ago

Sam Altman has in the past talked about gatekeeping AI compute to focus on important problems: things like solving for vaccines or intense science or even just things that will fundamentally improve humans day to day vs asking it dumb questions that could be found with a google search.

Havoc 2y ago

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. It’s the logical incremental step until an actual breakthrough is model on technique

GistNoesis 2y ago

>We need a Galileo and Kepler of AI. Let's imagine a modern day Galileo discover a new revolutionary algorithm that gives the same results for 1000x less compute. How would he transform this competitive advantage into some money without revealing the secret sauce ? Today's he will be in competition with people spending $10^8 to train. Even with the 10^3 cost reduction it's out of range for most people. And then he has to build a business around while keeping making sure his secret techniques don't leak. VCs and CEOs are just playing the controlling the business game, raising barrier to make sure they stay in control. Incremental advancements are great for them, because they get to stay in the lead while reducing the possibility that a revolutionary advancement comes to shuffle their game. Renaissance Technologies could maintain an edge over the whole stock market for plenty of years. AI-controlled company will probably be able too, but you won't hear about them.