4 ms·

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and loo

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild

ignoramous 4y ago
> Imagine when a dozen models are wired together... Wouldn't these models hallucinate more than normal, then?
causi 4y ago
I worry that the hardware requirements are only going to accelerate the cloud-OS integration. Imagine a PC that's entirely unusable offline.
- cj 4y ago
  > Imagine a PC that's entirely unusable offline. FWIW we had thin clients in computer labs in middle school / high school 15 years ago (and still today these are common in enterprise environments, e.g. Citrix). Biggest issue is network latency which is limited by the speed of light, so I imagine if computers in 10 years require resources not available locally it would likely be a local/cloud hybrid model.
ShamelessC 4y ago
Machine learning is actually premised on being “simple” to implement. The more priors you hardcode with clever algorithms, the closer you get to what we already have. The point is to automate the process of learning. We do this now with relatively simple loss functions and models containing relatively simple parameters. The main stipulation is that they are all defined to be continuous so that you can use the chain rule from calculus to calculate the error with respect to every parameter without taking so long that it would never finish. I agree that your suggested approach of applying cleverness to what we have now will probably produce better results. But that’s not going to stop better architectures, hardware and even entire regimes from being developed until we approach AGI. My suspicion is that there’s still a few breakthroughs waiting to be made. I also suspect that sufficiently advanced models will make such breakthroughs easier to discover.
xwdv 4y ago
People think something magical happens when AI are wired together and give each other feedback. Really you’re still just predicting the next word, but with extra steps.
- Teever 4y ago
  People think that something magical happens when transistors are wired together and give each other feedback. Really you're just switching switches on and off, but with extra steps.
ryneandal 4y ago
Personally, I'm less worried about AI than I am about what people using these models can do to others. Misinformation/disinformation, more believable scams, stuff like that.
quonn 4y ago
I have repeatedly argued against this notion of „just predicting the next word“. No. It‘s completing a conversation. It‘s true that it is doing this word by word, but it‘s kind of like saying a CNN is just predicting a label. Sure, but how? It‘s not doing it directly. It‘s doing it by recovering a lot of structure and in the end boiling that down to a label. Likewise a network trained to predict the next word may very well have worked out the whole sentence (implicitly, not as a text) in order to generate the next word.
DesiLurker 4y ago
I actually have high hopes for the hybrid architecture Ben Goertzel has been working on at OpenCog. I think the LLMs are soon going to hit a S curve w/o introduction of additional scientific knowledge like physics and notion of energy (wrt AGI development, they'll still be good for tonnes of other jobs displacing things).

ignoramous 4y ago

> Imagine when a dozen models are wired together... Wouldn't these models hallucinate more than normal, then?

causi 4y ago

I worry that the hardware requirements are only going to accelerate the cloud-OS integration. Imagine a PC that's entirely unusable offline.

cj 4y ago
> Imagine a PC that's entirely unusable offline. FWIW we had thin clients in computer labs in middle school / high school 15 years ago (and still today these are common in enterprise environments, e.g. Citrix). Biggest issue is network latency which is limited by the speed of light, so I imagine if computers in 10 years require resources not available locally it would likely be a local/cloud hybrid model.

ShamelessC 4y ago

Machine learning is actually premised on being “simple” to implement. The more priors you hardcode with clever algorithms, the closer you get to what we already have. The point is to automate the process of learning. We do this now with relatively simple loss functions and models containing relatively simple parameters. The main stipulation is that they are all defined to be continuous so that you can use the chain rule from calculus to calculate the error with respect to every parameter without taking so long that it would never finish. I agree that your suggested approach of applying cleverness to what we have now will probably produce better results. But that’s not going to stop better architectures, hardware and even entire regimes from being developed until we approach AGI. My suspicion is that there’s still a few breakthroughs waiting to be made. I also suspect that sufficiently advanced models will make such breakthroughs easier to discover.

xwdv 4y ago

People think something magical happens when AI are wired together and give each other feedback. Really you’re still just predicting the next word, but with extra steps.

Teever 4y ago
People think that something magical happens when transistors are wired together and give each other feedback. Really you're just switching switches on and off, but with extra steps.

ryneandal 4y ago

Personally, I'm less worried about AI than I am about what people using these models can do to others. Misinformation/disinformation, more believable scams, stuff like that.

quonn 4y ago

I have repeatedly argued against this notion of „just predicting the next word“. No. It‘s completing a conversation. It‘s true that it is doing this word by word, but it‘s kind of like saying a CNN is just predicting a label. Sure, but how? It‘s not doing it directly. It‘s doing it by recovering a lot of structure and in the end boiling that down to a label. Likewise a network trained to predict the next word may very well have worked out the whole sentence (implicitly, not as a text) in order to generate the next word.

DesiLurker 4y ago

I actually have high hopes for the hybrid architecture Ben Goertzel has been working on at OpenCog. I think the LLMs are soon going to hit a S curve w/o introduction of additional scientific knowledge like physics and notion of energy (wrt AGI development, they'll still be good for tonnes of other jobs displacing things).