4 ms·

We apparently have misaligned understandings of what we mean by "how they work." I agree, we don't know how to interpret the weight structure that the model lea

by lukev 4y ago

We apparently have misaligned understandings of what we mean by "how they work." I agree, we don't know how to interpret the weight structure that the model learns during training. But we do know exactly what happens mechanically during training and inference; what gets multiplied by what, what the inputs and outputs are, how data moves around the system. These are not some mysterious agents that could theoretically do or be anything, much less be secretly conscious (as a lot of alarmists are saying.) They are functions that multiply billions of numbers to generate output tokens. Their ability to output the "right" output tokens is not well understood, and nearly magical. That's what makes them so exciting.

famouswaffles 4y ago
It is all things considered pretty easy to set up GPT such that it runs on its own input forever while being able to interact with users/other systems. add an inner monologue/react and reflexion and you have a very powerful system. embody it with some physical locomotive machine and oh boy. no one has really put this all together yet but everything i've said has been done to some degree. The individual pieces are here. it's just a matter of time. I'm working on some such myself. What it could do is limited only by its intelligence (which is quite a bit higher than the base model as several papers have indicted) and the tools it controls (we seem to gladfully pile more and more control ). What it can be is...anything. If there's anything LLMs are good at, it's simulation. Even this system with thoughts we can theoretically configure to see would be difficult to control. theory and practicality would not meet the road. you will not be able to monitor this system in real time. We've seen bing (doesn't even have all i've described) take action when "upset". The only reason it didn't turn sour is because her actions are limited to search and ending the conversation. But that's obviously not the direction of things here. Can't say i want this train to stop. But i'm under no delusions it couldn't turn dangerous very quickly.
- lukev 4y ago
  I agree that LLMs could be one module in a future AGI system. I disagree that LLMs are good at simulation. They're good at prediction. They can only simulate to the degree that the thing they're simulating is present in their training data. Also, if you were trying to build an AGI, why would you NOT run it slowly at first so you could preserve and observe the logs? And if you wanted to build it to run full speed, why would you not build other single-purpose dumber AIs to watch it in case its thought stream diverged from expected behavior? There's a lot of escape hatches here.
  - famouswaffles 4y ago
    human.exe, robot.exe, malignant agent.exe are all very much simulations that an llm would have no problem running. >Also, if you were trying to build an AGI, why would you NOT run it slowly at first so you could preserve and observe the logs? I'm telling you that is extremely easy to do all the things i've said. Some might be interested in doing what you say. Others might not. at any rate, to be effective this requires real time monitoring of thoughts and actions. That's not feasible forever. an LLMs state can change. There's no guarantee the friendly agent you observed today will be friendly tomorrow. >And if you wanted to build it to run full speed, why would you not build other single-purpose dumber AIs to watch it in case its thought stream diverged from expected behavior? This is already done with say Bing. Not even remotely robust enough.

by lukev 4y ago

famouswaffles 4y ago

It is all things considered pretty easy to set up GPT such that it runs on its own input forever while being able to interact with users/other systems. add an inner monologue/react and reflexion and you have a very powerful system. embody it with some physical locomotive machine and oh boy. no one has really put this all together yet but everything i've said has been done to some degree. The individual pieces are here. it's just a matter of time. I'm working on some such myself. What it could do is limited only by its intelligence (which is quite a bit higher than the base model as several papers have indicted) and the tools it controls (we seem to gladfully pile more and more control ). What it can be is...anything. If there's anything LLMs are good at, it's simulation. Even this system with thoughts we can theoretically configure to see would be difficult to control. theory and practicality would not meet the road. you will not be able to monitor this system in real time. We've seen bing (doesn't even have all i've described) take action when "upset". The only reason it didn't turn sour is because her actions are limited to search and ending the conversation. But that's obviously not the direction of things here. Can't say i want this train to stop. But i'm under no delusions it couldn't turn dangerous very quickly.

lukev 4y ago
I agree that LLMs could be one module in a future AGI system. I disagree that LLMs are good at simulation. They're good at prediction. They can only simulate to the degree that the thing they're simulating is present in their training data. Also, if you were trying to build an AGI, why would you NOT run it slowly at first so you could preserve and observe the logs? And if you wanted to build it to run full speed, why would you not build other single-purpose dumber AIs to watch it in case its thought stream diverged from expected behavior? There's a lot of escape hatches here.
- famouswaffles 4y ago
  human.exe, robot.exe, malignant agent.exe are all very much simulations that an llm would have no problem running. >Also, if you were trying to build an AGI, why would you NOT run it slowly at first so you could preserve and observe the logs? I'm telling you that is extremely easy to do all the things i've said. Some might be interested in doing what you say. Others might not. at any rate, to be effective this requires real time monitoring of thoughts and actions. That's not feasible forever. an LLMs state can change. There's no guarantee the friendly agent you observed today will be friendly tomorrow. >And if you wanted to build it to run full speed, why would you not build other single-purpose dumber AIs to watch it in case its thought stream diverged from expected behavior? This is already done with say Bing. Not even remotely robust enough.