5 ms·

It's clear we're searching for the god algorithm of AI, just like physicists are searching for theory of everything. Are transformers the answer though?

mirekrusin 3y ago
If they were we'd stop searching already, no?
goldenkey 3y ago
There's really no God algorithm needed, just something good enough to assist with research of the next tier of hardware, energy, and code for AI.
Workaccount2 3y ago
It seems more likely that there will be multiple avenues to AGI, all with their strengths and weaknesses. But perhaps the "God AI" will be a multifaceted model composed of many different models acting in unison.
- falcor84 3y ago
  Exactly, just as the true god has seven aspects [0]. [0] https://en.wikipedia.org/wiki/Themes_in_A_Song_of_Ice_and_Fire#Faith_of_the_Seven https://en.wikipedia.org/wiki/Themes_in_A_Song_of_Ice_and_Fi...
  - bee_rider 3y ago
    On the other hand, most of the original gods were parts of polytheistic pantheons. Maybe a bunch of models that represent identities and biases could be more useful, they could argue amongst themselves, presenting a more full point of view, users could become familiar with the particular perspectives.
- kstrauser 3y ago
  I could see something like the "modularity of mind" model of human consciousness, where multiple approaches are working on "subconscious" solutions to a given problem in parallel, with a top layer deciding which is appropriate at the moment. My human brain doesn't use the same algorithm for learning to play a song on a piano as learning to play a new board game. I'm not an AI person, but it seems reasonable to imagine we'd have different "modules" to apply as needed. AlphaGo probably sucks at conversation. ChatGPT can't play Go. The part of my brain writing this couldn't throw a baseball. The physics engine that lets me throw a baseball couldn't write this. Is there a reason we'd want or need one specific AI approach to be universally applicable?
  - platz 3y ago
    The Bicameral Mind
- two_in_one 3y ago
  Most likely there will be more then one specimen claiming to be AGI. From different groups. And they will be hard to compare.
theptip 3y ago
No, not at all. Just looking for the next step of many, stacking S-curves atop each other.
mikewarot 3y ago
We're searching for an efficient algorithm that leads to AGI. Given sufficient time and compute, I'm sure that we could get there with existing stuff, by accident, and we wouldn't realize it before moving on to the next thing... and there'd be a poor orphan AGI, lost in a Git repo, waiting for runtime.
- sfink 3y ago
  "Given sufficient time and compute" covers up a lot, though. The ultimate God AGI that would be created through that sort of process would take the form of a large room filled with a whole lot of monkeys and typewriters.
lossolo 3y ago
As it stands, each token processed by transformer requires a constant amount of computation and energy. For an AGI system, this would imply the ability to solve problems of any complexity with a fixed amount of energy. But if this were true, it would essentially mean that P equals NP, a major theoretical breakthrough in computational complexity theory. IMO we are still missing something.
AndrewKemendo 3y ago
Algorithm is too simple, but yes - what does the system look like I think the transformers architecture, or something very similar with eventually-on-policy time series forecasting in a markov decision process, is the right answer actually and was what I have been trying to make progress on for a long time[1]. [1]https://kemendo.com/research/streaminference.html https://kemendo.com/research/streaminference.html
- snovv_crash 3y ago
  That plus some Monte Carlo search as in AlphaZero makes a very strong (although computationally heavy) contender for 'alive' AI.
  - AndrewKemendo 3y ago
    After this past few years of progress on multivariate stream forecasting I really think it’s going to be different implementations of the same basic streaming data forecasting architecture
amelius 3y ago
Theory of everything is supposed to be beautiful. Theory of AI is just going to be some weird network with a shit-ton of compute power, where the latter is more important to the outcome than the former.
skinner_ 3y ago
There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906 https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.
MagicMoonlight 3y ago
No, it’s a much simpler architecture. Anything that evolved can’t require a really complicated niche structure. The difficulty is the scale. Every synapse of a neuron is effectively a neuron itself, and every synapse acts on the synapses around it. So before you’ve even got to the neuron as a whole you’ve already got the equivalent of thousands of neurons and logic gates. Then the final result gets passed on to thousands more neurons. I don’t know how you would recreate such complexity in programming. It’s not just the scale, it’s the flexibility of the structure.

mirekrusin 3y ago

If they were we'd stop searching already, no?

goldenkey 3y ago

There's really no God algorithm needed, just something good enough to assist with research of the next tier of hardware, energy, and code for AI.

Workaccount2 3y ago

It seems more likely that there will be multiple avenues to AGI, all with their strengths and weaknesses. But perhaps the "God AI" will be a multifaceted model composed of many different models acting in unison.

falcor84 3y ago
Exactly, just as the true god has seven aspects [0]. [0] https://en.wikipedia.org/wiki/Themes_in_A_Song_of_Ice_and_Fire#Faith_of_the_Seven https://en.wikipedia.org/wiki/Themes_in_A_Song_of_Ice_and_Fi...
- bee_rider 3y ago
  On the other hand, most of the original gods were parts of polytheistic pantheons. Maybe a bunch of models that represent identities and biases could be more useful, they could argue amongst themselves, presenting a more full point of view, users could become familiar with the particular perspectives.
kstrauser 3y ago
I could see something like the "modularity of mind" model of human consciousness, where multiple approaches are working on "subconscious" solutions to a given problem in parallel, with a top layer deciding which is appropriate at the moment. My human brain doesn't use the same algorithm for learning to play a song on a piano as learning to play a new board game. I'm not an AI person, but it seems reasonable to imagine we'd have different "modules" to apply as needed. AlphaGo probably sucks at conversation. ChatGPT can't play Go. The part of my brain writing this couldn't throw a baseball. The physics engine that lets me throw a baseball couldn't write this. Is there a reason we'd want or need one specific AI approach to be universally applicable?
- platz 3y ago
  The Bicameral Mind
two_in_one 3y ago
Most likely there will be more then one specimen claiming to be AGI. From different groups. And they will be hard to compare.

theptip 3y ago

No, not at all. Just looking for the next step of many, stacking S-curves atop each other.

mikewarot 3y ago

We're searching for an efficient algorithm that leads to AGI. Given sufficient time and compute, I'm sure that we could get there with existing stuff, by accident, and we wouldn't realize it before moving on to the next thing... and there'd be a poor orphan AGI, lost in a Git repo, waiting for runtime.

sfink 3y ago
"Given sufficient time and compute" covers up a lot, though. The ultimate God AGI that would be created through that sort of process would take the form of a large room filled with a whole lot of monkeys and typewriters.

lossolo 3y ago

As it stands, each token processed by transformer requires a constant amount of computation and energy. For an AGI system, this would imply the ability to solve problems of any complexity with a fixed amount of energy. But if this were true, it would essentially mean that P equals NP, a major theoretical breakthrough in computational complexity theory. IMO we are still missing something.

AndrewKemendo 3y ago

Algorithm is too simple, but yes - what does the system look like I think the transformers architecture, or something very similar with eventually-on-policy time series forecasting in a markov decision process, is the right answer actually and was what I have been trying to make progress on for a long time[1]. [1]https://kemendo.com/research/streaminference.html https://kemendo.com/research/streaminference.html

snovv_crash 3y ago
That plus some Monte Carlo search as in AlphaZero makes a very strong (although computationally heavy) contender for 'alive' AI.
- AndrewKemendo 3y ago
  After this past few years of progress on multivariate stream forecasting I really think it’s going to be different implementations of the same basic streaming data forecasting architecture

amelius 3y ago

Theory of everything is supposed to be beautiful. Theory of AI is just going to be some weird network with a shit-ton of compute power, where the latter is more important to the outcome than the former.

skinner_ 3y ago

There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906 https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.

MagicMoonlight 3y ago

No, it’s a much simpler architecture. Anything that evolved can’t require a really complicated niche structure. The difficulty is the scale. Every synapse of a neuron is effectively a neuron itself, and every synapse acts on the synapses around it. So before you’ve even got to the neuron as a whole you’ve already got the equivalent of thousands of neurons and logic gates. Then the final result gets passed on to thousands more neurons. I don’t know how you would recreate such complexity in programming. It’s not just the scale, it’s the flexibility of the structure.