4 ms·

Why would its source code be part of the training corpus? Anything is possible, but it could just as easily be making this up Edit: actually I want to expand a

Why would its source code be part of the training corpus? Anything is possible, but it could just as easily be making this up Edit: actually I want to expand a little bit. This model has had a lot of work put in to ensure "safety" but only in a shallow sense. Real failure modes, like being unable to tell if it is lying about stuff, are ignored. Whether or not it for some reason has it's own source code memorized as part of the weights (which would really be weird), this is definitely a failure because it provides output that cannot be trusted.

jacquesm 4y ago
It could be, but it appears to be consistent across several runs.
- orbital-decay 4y ago
  It also doesn't mean anything - could be just entirely made up stuff from some metastable point in its space it converged on for your question. The model is really large and coherent, but it's not capable of drawing any line between facts and fiction, which makes it a perfect bullshitter. This model is unable to follow any sort of logic, so it usually takes any nonsense question at face value and tries to answer it honestly, even if it means inventing very plausible lies in the process. Unless it goes against its biases of course - in that case they take preference. This is enough for it to sound very confident, which can trick most people. (see also the warning in my profile)
  - ugurnot 4y ago
    Having a consistent output to same input is something. I have done a couple of experiments starting with same sentence and each time it was a little bit nuanced. I would be interested if the answer is the same for a targeted question. Still, it would be better not to draw conclusions early.
    - orbital-decay 4y ago
      Most likely their repo is similar to hundreds of others, so it probably just gives the structure of some typical ML code (which is imprinted in the model really well), skewed towards the initial request.

jacquesm 4y ago

It could be, but it appears to be consistent across several runs.

orbital-decay 4y ago
It also doesn't mean anything - could be just entirely made up stuff from some metastable point in its space it converged on for your question. The model is really large and coherent, but it's not capable of drawing any line between facts and fiction, which makes it a perfect bullshitter. This model is unable to follow any sort of logic, so it usually takes any nonsense question at face value and tries to answer it honestly, even if it means inventing very plausible lies in the process. Unless it goes against its biases of course - in that case they take preference. This is enough for it to sound very confident, which can trick most people. (see also the warning in my profile)
- ugurnot 4y ago
  Having a consistent output to same input is something. I have done a couple of experiments starting with same sentence and each time it was a little bit nuanced. I would be interested if the answer is the same for a targeted question. Still, it would be better not to draw conclusions early.
  - orbital-decay 4y ago
    Most likely their repo is similar to hundreds of others, so it probably just gives the structure of some typical ML code (which is imprinted in the model really well), skewed towards the initial request.