7 ms·

I'm going to tap the sign again: - [X] Text - [X] Images - [X] Audio - [ ] Videos (in progress) - [ ] 3D Meshes and Textures (in progress) - [ ] Ge

I'm going to tap the sign again: - [X] Text - [X] Images - [X] Audio - [ ] Videos (in progress) - [ ] 3D Meshes and Textures (in progress) - [ ] Genetics (in progress) - [ ] Physics Simulation (in progress) - [ ] Mathematics - [ ] Logic and Algorithms aka Planning and Optimization - [ ] Reasoning - [ ] Emotion - [ ] Consciousness We still have a lot of data to crunch but it's not nearly enough so we're also going to have to collect and generate a lot more of it. Some of these items require data that we don't even know how to collect yet. Barring some kind of disastrous event, draconian regulation, or politically/culturally motivated demonization of ML I don't see GPU demand dropping any time soon.

danielmarkbruce 2y ago
Plus we will re-crunch it a bajillion times and run inference a bajillion times.
pipo234 2y ago
Extrapolation is dangerous. People tend to overestimate what's possible in a year while underestimating the possibilities next decade. So far we're mostly seeing huge investments hoping for a short term goals while we're not sure whether long term goals (like: Logic and Algorithms aka Planning and Optimization) would even benefit from more compute. Maybe, yes. Shedding some hindsight on earlier extrapolations — The billions pored into the metaverse or self-driving didn't yield the results we expected in the period we expected.
- throwaway4aday 2y ago
  While I agree that we don't want to extrapolate too much I disagree that this type of exploration may not benefit from more compute. We won't know until we try and since we have what seems to be a very generalizable architecture it makes sense to take the brute force approach of creating models of that data by scaling the amount of data and the amount of compute we dedicate to it. If it turns out not to work then we've learned something. As it turns out, Logic and Algorithms has seen some early success using Transformers (Searchformer) https://arxiv.org/abs/2402.14083 https://arxiv.org/abs/2402.14083
mschuster91 2y ago
I'd also add protein folding and interactions to the list of "pretty much solved" after AlphaFold 2 [1]. [1] https://deepmind.google/technologies/alphafold/ https://deepmind.google/technologies/alphafold/
jononor 2y ago
The virtual world parts such as video and 3d, and plausible physics, I think we are going to do as well as image/audio/text within the next 10 years. Maybe even 5. These things primarily need to be believable and mostly-not-directly-wrong to serve a lot of usecases. And the way to get it to that level seems to be "just train it on Internet scale amounts of data". Whether we will really have cracked the physical world connections, of physics, genetics, etc that we can use it to make physical products, changes etc I am less sure. Many usecases like medicine require not just correctness, but also a degree of verifiability. It is being worked on a lot, with many promising results. But the just-scale-the-training data strategy seems less viable here, both because relevant data is less prevalent and may not give the level of correctness.
user432678 2y ago
“Completely autonomous self-driving cars next year”, — every self-driving startup CEO in 2015. As someone said in the comments above, it’s a miserable way of life, but I’m still very pessimistic about this extrapolation.
- fragmede 2y ago
  We can quibble where Waymo falls on the "completely autonomous" scale of things, but self driving cars are here, 9 years after 2015.
  - user432678 2y ago
    Oh, sorry for my non-US-centric view. And yes, I am nitpicking, sort of.
- throwaway4aday 2y ago
  I didn't say anything about a timeline for _solving_ these, just that the short timeline for drop off in demand for compute is unfounded since there is still so much ground to cover. The article takes the shortsighted view that the current state of text generation feels like it's in a lull (I strongly disagree with this for a variety of reasons chief among them being that 1. the supposed stall in progress hasn't gone on long enough to call it and 2. the big players are all focusing on productization and making the current SOTA as cheap as possible to improve their bottom line and expand its applicability) but there are a large number of other domains and sub-domains where these techniques can be applied and will likely see similar rapid advances as the amount of available data increases.
Yizahi 2y ago
Some of these are not like the others

danielmarkbruce 2y ago

Plus we will re-crunch it a bajillion times and run inference a bajillion times.

pipo234 2y ago

Extrapolation is dangerous. People tend to overestimate what's possible in a year while underestimating the possibilities next decade. So far we're mostly seeing huge investments hoping for a short term goals while we're not sure whether long term goals (like: Logic and Algorithms aka Planning and Optimization) would even benefit from more compute. Maybe, yes. Shedding some hindsight on earlier extrapolations — The billions pored into the metaverse or self-driving didn't yield the results we expected in the period we expected.

throwaway4aday 2y ago
While I agree that we don't want to extrapolate too much I disagree that this type of exploration may not benefit from more compute. We won't know until we try and since we have what seems to be a very generalizable architecture it makes sense to take the brute force approach of creating models of that data by scaling the amount of data and the amount of compute we dedicate to it. If it turns out not to work then we've learned something. As it turns out, Logic and Algorithms has seen some early success using Transformers (Searchformer) https://arxiv.org/abs/2402.14083 https://arxiv.org/abs/2402.14083

mschuster91 2y ago

I'd also add protein folding and interactions to the list of "pretty much solved" after AlphaFold 2 [1]. [1] https://deepmind.google/technologies/alphafold/ https://deepmind.google/technologies/alphafold/

jononor 2y ago

The virtual world parts such as video and 3d, and plausible physics, I think we are going to do as well as image/audio/text within the next 10 years. Maybe even 5. These things primarily need to be believable and mostly-not-directly-wrong to serve a lot of usecases. And the way to get it to that level seems to be "just train it on Internet scale amounts of data". Whether we will really have cracked the physical world connections, of physics, genetics, etc that we can use it to make physical products, changes etc I am less sure. Many usecases like medicine require not just correctness, but also a degree of verifiability. It is being worked on a lot, with many promising results. But the just-scale-the-training data strategy seems less viable here, both because relevant data is less prevalent and may not give the level of correctness.

user432678 2y ago

“Completely autonomous self-driving cars next year”, — every self-driving startup CEO in 2015. As someone said in the comments above, it’s a miserable way of life, but I’m still very pessimistic about this extrapolation.

fragmede 2y ago
We can quibble where Waymo falls on the "completely autonomous" scale of things, but self driving cars are here, 9 years after 2015.
- user432678 2y ago
  Oh, sorry for my non-US-centric view. And yes, I am nitpicking, sort of.
throwaway4aday 2y ago
I didn't say anything about a timeline for _solving_ these, just that the short timeline for drop off in demand for compute is unfounded since there is still so much ground to cover. The article takes the shortsighted view that the current state of text generation feels like it's in a lull (I strongly disagree with this for a variety of reasons chief among them being that 1. the supposed stall in progress hasn't gone on long enough to call it and 2. the big players are all focusing on productization and making the current SOTA as cheap as possible to improve their bottom line and expand its applicability) but there are a large number of other domains and sub-domains where these techniques can be applied and will likely see similar rapid advances as the amount of available data increases.

Yizahi 2y ago

Some of these are not like the others