3 ms·

My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass. And then t

by pstuart 4mo ago

My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass. And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.

orbital-decay 4mo ago
I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation. In other words, yes it might be possible it experiences something in its own bizarre timeline and world, for some definitions of "experiencing". At least it developed primitive circuitry functionally equivalent to biological systems. But "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.
morpheos137 4mo ago
ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?
- orbital-decay 4mo ago
  This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2 https://arxiv.org/abs/2309.10668v2
  - morpheos137 4mo ago
    again, try thinking critically it is not merely an interpolator means it can interpolate on many dimensions. it does not follow that greater than human capability results from doing so. explain to me how a statistical function approximator (which is what a transformer is) with human training input and human tuning (rhlf) exceeds the aggregate human cognitive envelope? What is the mechanism? Let's say an LLM makes an inference that no human could have possibly made (arguably impossible itself) how does the inference survive rhlf or become useful to humans if they can not judge its validity? how do you take the shape of the human corpus and all its gradients and some how arrive at something greater than human, where was the missing information hiding?
    - orbital-decay 4mo ago
      Sorry, I just noticed I posted a wrong link in the comment above. Here's the proper one: https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485
      - morpheos137 4mo ago
        ok however i would say extrapolating the current data set is not a way to exceed the the human envelope. it is unclear to me the human evelope has been demonstrated as a convex hull or how transformers could find points outside it. in other words intelligence and knowledge does not exist as some abstract possibility space but only as a set of contextual contingences. LLMs have no context beyond the human envelope. weights are frozen. there is no selection mechanism for retaining suprahuman inferences made during training if that were even possible. thus i grant that llms. could theoretically make inferences outside the human corpus there is no way to distinguish the from errors or hallucinations during training (because by definition the are beyond human capacity) and no iterative learning from experience process after training (frozen weights). thus it seems impossible for today's models to exceed aggregate human capacity.
    - Timwi 4mo ago
      > how do you take the shape of the human corpus and all its gradients and [somehow] arrive at something greater than human, where was the missing information hiding? Well, how do humans do it? Scientists discover new stuff that isn't in any corpus. Even I as a lowly computer user occasionally figure something out about a software without reading a help screen. It's obviously possible to arrive at new information by interpolating existing information.

by pstuart 4mo ago

orbital-decay 4mo ago

I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation. In other words, yes it might be possible it experiences something in its own bizarre timeline and world, for some definitions of "experiencing". At least it developed primitive circuitry functionally equivalent to biological systems. But "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.

morpheos137 4mo ago

ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?

orbital-decay 4mo ago
This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2 https://arxiv.org/abs/2309.10668v2
- morpheos137 4mo ago
  again, try thinking critically it is not merely an interpolator means it can interpolate on many dimensions. it does not follow that greater than human capability results from doing so. explain to me how a statistical function approximator (which is what a transformer is) with human training input and human tuning (rhlf) exceeds the aggregate human cognitive envelope? What is the mechanism? Let's say an LLM makes an inference that no human could have possibly made (arguably impossible itself) how does the inference survive rhlf or become useful to humans if they can not judge its validity? how do you take the shape of the human corpus and all its gradients and some how arrive at something greater than human, where was the missing information hiding?
  - orbital-decay 4mo ago
    Sorry, I just noticed I posted a wrong link in the comment above. Here's the proper one: https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485
    - morpheos137 4mo ago
      ok however i would say extrapolating the current data set is not a way to exceed the the human envelope. it is unclear to me the human evelope has been demonstrated as a convex hull or how transformers could find points outside it. in other words intelligence and knowledge does not exist as some abstract possibility space but only as a set of contextual contingences. LLMs have no context beyond the human envelope. weights are frozen. there is no selection mechanism for retaining suprahuman inferences made during training if that were even possible. thus i grant that llms. could theoretically make inferences outside the human corpus there is no way to distinguish the from errors or hallucinations during training (because by definition the are beyond human capacity) and no iterative learning from experience process after training (frozen weights). thus it seems impossible for today's models to exceed aggregate human capacity.
  - Timwi 4mo ago
    > how do you take the shape of the human corpus and all its gradients and [somehow] arrive at something greater than human, where was the missing information hiding? Well, how do humans do it? Scientists discover new stuff that isn't in any corpus. Even I as a lowly computer user occasionally figure something out about a software without reading a help screen. It's obviously possible to arrive at new information by interpolating existing information.