3 ms·

The main issue with this argument that current AI models are extremely inefficient. Models evaluate all weights on every pass, LLMs recall their entirety of kno

The main issue with this argument that current AI models are extremely inefficient. Models evaluate all weights on every pass, LLMs recall their entirety of knowledge just to output a part of a word and then do it all again. Often a hand crafted algorithm can achieve what a neural network can in a small fraction of compute. There is likely a 1e4x-1e6x compute gap that can be achieved with the right algorithm, maybe even a 1e10x and once we reach self improvement that gap could close very fast. They say that the human brain has 30 PFLOPS of compute power and this is sometimes used as a reference for how much compute is needed for intelligence, but that completely misses the point: The human brain is extremely inefficient. Symbolic mathematics can be unreasonably [1] effective at making accurate predictions about the future with very little computation, but many of these solutions are inaccessible to evolution. Once AI learns to construct efficient mathematical models of the universe, the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. [1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

rini17 3y ago
Before the first AI winter there was incredible effort in leveraging symbolical approach. It did deliver many useful things but far from hype. For example we still have no self-repairing systems, only specific fault-tolerant algoritms but by and large software is as fragile as ever. You can't avoid considering everything when trying to deal with unexpected data/issue. Only in hidsight you can then say "wow that was sooo inefficient".
- WithinReason 3y ago
  A calculator can compute the 5th root of 1692662195786551 much more efficiently than a person
bob1029 3y ago
> the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. I am in the camp that this is possible and that humans have achieved it already. I believe things are going to get very weird when these techniques become mainstream. The GPU-cluster-bound algorithms are just a tiny stepping stone. Do we start taking computers away from people? What do we do about this if it turns out the only barrier to entry is the software?
- ChatGTP 3y ago
  I’d say your going to get “AI” rammed so far down your throat it might as well come out of your ass before anyone is going to let their puters be taken away.
  - pcthrowaway 3y ago
    Maybe our desktops and laptops get taken away, and they leave us with our smartphones (which are locked down for our safety). 90% of people wouldn't give a shit as long as they have their smartphones
    - ChatGTP 3y ago
      It's true
jillesvangurp 3y ago
It's not an issue, it's an optimization problem. I look at this as an exponential. Before chat gpt was launched a bit over a year ago, the amount of AI in our lives was relatively low. We had bits and pieces here and there but everyday life wasn't really affected by it. At best you might be yelling at siri or alexa to do whatever and maybe getting some OKish results. Fast forward 1 year. We now have students revolutionizing education (by submitting AI doctored papers and homework that isn't half bad). People in law dealing with people using AI. AIs passing medical, legal, and other exams. That was just one year. And yes, it's tediously slow to use. But last year we had nothing. It's going to be an exponential in terms of adoption, response speed, energy usage, parameter numbers, context size and a lot of optimization. I think we are not that far off from AIs being able to keep up with a conversation and being able to respond right away. For all you hitchhiker's to the galaxy readers, that would be babelfish sorted. Translation quality is already pretty awesome with chat gpt 4. I've not caught it making major translation mistakes. Speech to text, translate the text, text to speech. Now we are conversing with our chat bots rather than hammering out sentences in some text box. It's hard to not see that go from a novelty "oh this is cool" to world + dog basically using this on a daily basis throughout the day. At the same time it will get smarter. Everybody is waiting for the singularity where it is clearly smarter than Einstein in his prime and running circles around everyone else. But for most questions you might ask chat gpt and get a reasonable answer, the vast majority of people around you might be pretty useless. Is that smarter? I don't know. But definitely more useful. From where we are now to this being ubiquitous and used by pretty much anyone is probably a few years. I remember when smart phones happened. One day everybody was minding their own business and a year later the streets were full of zombies glued to their screen. Just happened very quickly. This might be the same but possibly quicker.
audunw 3y ago
> The human brain is extremely inefficient. That depends heavily on what kind of task we're talking about. The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach. I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is. > Once AI learns to construct efficient mathematical models of the universe There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)
rafaelero 3y ago
> Models evaluate all weights on every pass Not true for GPT-4.

rini17 3y ago

Before the first AI winter there was incredible effort in leveraging symbolical approach. It did deliver many useful things but far from hype. For example we still have no self-repairing systems, only specific fault-tolerant algoritms but by and large software is as fragile as ever. You can't avoid considering everything when trying to deal with unexpected data/issue. Only in hidsight you can then say "wow that was sooo inefficient".

WithinReason 3y ago
A calculator can compute the 5th root of 1692662195786551 much more efficiently than a person

bob1029 3y ago

> the amount of computation available today in a PC could be enough to do things (good and bad) way beyond human ability. I am in the camp that this is possible and that humans have achieved it already. I believe things are going to get very weird when these techniques become mainstream. The GPU-cluster-bound algorithms are just a tiny stepping stone. Do we start taking computers away from people? What do we do about this if it turns out the only barrier to entry is the software?

ChatGTP 3y ago
I’d say your going to get “AI” rammed so far down your throat it might as well come out of your ass before anyone is going to let their puters be taken away.
- pcthrowaway 3y ago
  Maybe our desktops and laptops get taken away, and they leave us with our smartphones (which are locked down for our safety). 90% of people wouldn't give a shit as long as they have their smartphones
  - ChatGTP 3y ago
    It's true

jillesvangurp 3y ago

It's not an issue, it's an optimization problem. I look at this as an exponential. Before chat gpt was launched a bit over a year ago, the amount of AI in our lives was relatively low. We had bits and pieces here and there but everyday life wasn't really affected by it. At best you might be yelling at siri or alexa to do whatever and maybe getting some OKish results. Fast forward 1 year. We now have students revolutionizing education (by submitting AI doctored papers and homework that isn't half bad). People in law dealing with people using AI. AIs passing medical, legal, and other exams. That was just one year. And yes, it's tediously slow to use. But last year we had nothing. It's going to be an exponential in terms of adoption, response speed, energy usage, parameter numbers, context size and a lot of optimization. I think we are not that far off from AIs being able to keep up with a conversation and being able to respond right away. For all you hitchhiker's to the galaxy readers, that would be babelfish sorted. Translation quality is already pretty awesome with chat gpt 4. I've not caught it making major translation mistakes. Speech to text, translate the text, text to speech. Now we are conversing with our chat bots rather than hammering out sentences in some text box. It's hard to not see that go from a novelty "oh this is cool" to world + dog basically using this on a daily basis throughout the day. At the same time it will get smarter. Everybody is waiting for the singularity where it is clearly smarter than Einstein in his prime and running circles around everyone else. But for most questions you might ask chat gpt and get a reasonable answer, the vast majority of people around you might be pretty useless. Is that smarter? I don't know. But definitely more useful. From where we are now to this being ubiquitous and used by pretty much anyone is probably a few years. I remember when smart phones happened. One day everybody was minding their own business and a year later the streets were full of zombies glued to their screen. Just happened very quickly. This might be the same but possibly quicker.

audunw 3y ago

> The human brain is extremely inefficient. That depends heavily on what kind of task we're talking about. The human brain uses just 12 watts, and with that it can still perform certain tasks that a computer using 100 times as much power can't even approach. I get the point you were trying to make. In certain tasks it's extremely inefficient, yes. But in general, it's still crazy how power efficient it is. > Once AI learns to construct efficient mathematical models of the universe There's not going to be just one model. Whether an AI (in the short term) can be more efficient than a human at discovering these efficient mathematical models remains to be seen. It seems like the creativity and discovery needed to do this is exactly what neural nets can be good at, but then you're back to something fairly inefficient (with todays algorithms)

rafaelero 3y ago

> Models evaluate all weights on every pass Not true for GPT-4.