5 ms·

Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what

by svsoc 11mo ago

Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.

jandrewrogers 11mo ago
Optimal learning is an interesting problem in computer science because it is fundamentally bound by geometric space complexity rather than computational complexity. You can bend the curve but the approximations degrade rapidly and still have a prohibitively expensive exponential space complexity. We have literature for this; a lot of the algorithmic information theory work in AI was about characterizing these limits. The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits. This was thoroughly studied by the US defense community decades ago. The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers. I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency. Personally, I find these “fundamental limits of computation” things to be extremely fascinating.
- davidivadavid 11mo ago
  Any reference material (papers/textbooks) on that topic? It does sound fun.
- hunterpayne 11mo ago
  So I studied Machine Learning too. One of the main things I learned is that for any problem there is an ideally sized model that when trained will produce the lowest error rate. Now, when you do multi-class learning (training a model for multiple problems), that ideally sized model is larger but there is still an optimum sized model. Seems to me that for GAI, there will also be an ideally sized model. I wouldn't be surprised if the complexity of that model was very similar to the size of the human brain. If that is the case, then some sort of super-intelligence isn't possible in any meaningful way. This would seem to track with what we are seeing in the today's LLMs. When they build bigger models, they often don't perform as well as the previous one which perhaps was at some maximum/ideal complexity. I suspect, we will continue to run into this barrier over and over again.
  - czl 11mo ago
    > for any problem there is an ideally sized model that when trained will produce the lowest error rate. You studied ML before discovery of "double descent"? https://youtu.be/z64a7USuGX0 https://youtu.be/z64a7USuGX0
    - hunterpayne 11mo ago
      I did, however I have also observed ideal sized models since then in algorithms designed with knowledge of it.
adastra22 11mo ago
Not the person you are responding to, but much of the conclusions drawn by Bostrom (and most of EY’s ideas are credited to Bostrom) depend on infinities. The orthogonality thesis being series from AIXI, for example. EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.
- pas 11mo ago
  As far as I understand ChatGPT is not capable of self-improvement, so EY's argument is not applicable to it. (At least based on this https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf from 2013.) The FOOM argument starts with some kind of goal-directed agent (that escapes and then it) starts building a more capable version of itself (and then goal drift might set in might not) If you tell ChatGPT to build ChatGPT++ and leave currently there's no time horizon within it would accomplish either that or escape, or anything, because now it gives you tokens rendered on some website. The argument is not that AI is a magic box. - The argument is that if there's a process that improves AI. [1] - And if during that process AI becomes so capable that it can materially contribute to the process, and eventually continue (un)supervised. [2] - Then eventually it'll escape and do whatever it wants, and then eventually the smallest misalignment means we become expendable resources. I think the argument might be valid logically, but the constant factors are very important to the actual meaning and obviously we don't know them. (But the upper and lower estimates are far. Hence the whole debate.) [1] Look around, we have a process that's like that. However gamed and flawed we have METR scores and ARC-AGI benchmarks, and thousands of really determined and skillful people working on it, hundreds of billions of capital deployed to keep this process going. [2] We are not there yet, but decades after peak oil arguments we are very good at drawing various hockey stick curves.
  - 11mo ago

by svsoc 11mo ago

Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.

jandrewrogers 11mo ago

Optimal learning is an interesting problem in computer science because it is fundamentally bound by geometric space complexity rather than computational complexity. You can bend the curve but the approximations degrade rapidly and still have a prohibitively expensive exponential space complexity. We have literature for this; a lot of the algorithmic information theory work in AI was about characterizing these limits. The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits. This was thoroughly studied by the US defense community decades ago. The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers. I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency. Personally, I find these “fundamental limits of computation” things to be extremely fascinating.

davidivadavid 11mo ago
Any reference material (papers/textbooks) on that topic? It does sound fun.
hunterpayne 11mo ago
So I studied Machine Learning too. One of the main things I learned is that for any problem there is an ideally sized model that when trained will produce the lowest error rate. Now, when you do multi-class learning (training a model for multiple problems), that ideally sized model is larger but there is still an optimum sized model. Seems to me that for GAI, there will also be an ideally sized model. I wouldn't be surprised if the complexity of that model was very similar to the size of the human brain. If that is the case, then some sort of super-intelligence isn't possible in any meaningful way. This would seem to track with what we are seeing in the today's LLMs. When they build bigger models, they often don't perform as well as the previous one which perhaps was at some maximum/ideal complexity. I suspect, we will continue to run into this barrier over and over again.
- czl 11mo ago
  > for any problem there is an ideally sized model that when trained will produce the lowest error rate. You studied ML before discovery of "double descent"? https://youtu.be/z64a7USuGX0 https://youtu.be/z64a7USuGX0
  - hunterpayne 11mo ago
    I did, however I have also observed ideal sized models since then in algorithms designed with knowledge of it.

adastra22 11mo ago

Not the person you are responding to, but much of the conclusions drawn by Bostrom (and most of EY’s ideas are credited to Bostrom) depend on infinities. The orthogonality thesis being series from AIXI, for example. EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.

pas 11mo ago
As far as I understand ChatGPT is not capable of self-improvement, so EY's argument is not applicable to it. (At least based on this https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf from 2013.) The FOOM argument starts with some kind of goal-directed agent (that escapes and then it) starts building a more capable version of itself (and then goal drift might set in might not) If you tell ChatGPT to build ChatGPT++ and leave currently there's no time horizon within it would accomplish either that or escape, or anything, because now it gives you tokens rendered on some website. The argument is not that AI is a magic box. - The argument is that if there's a process that improves AI. [1] - And if during that process AI becomes so capable that it can materially contribute to the process, and eventually continue (un)supervised. [2] - Then eventually it'll escape and do whatever it wants, and then eventually the smallest misalignment means we become expendable resources. I think the argument might be valid logically, but the constant factors are very important to the actual meaning and obviously we don't know them. (But the upper and lower estimates are far. Hence the whole debate.) [1] Look around, we have a process that's like that. However gamed and flawed we have METR scores and ARC-AGI benchmarks, and thousands of really determined and skillful people working on it, hundreds of billions of capital deployed to keep this process going. [2] We are not there yet, but decades after peak oil arguments we are very good at drawing various hockey stick curves.
- 11mo ago