4 ms·

I think the hard / unknown part is how you know you’ve identified all of the failure modes that need to be tested. Tests of humans have evolved over a long tim

I think the hard / unknown part is how you know you’ve identified all of the failure modes that need to be tested. Tests of humans have evolved over a long time and large sample size, and humans may be more similar to each other than LLMs are, so failure modes may be more universal. But very short history, small sample size, and diversity of architecture and training means we really don’t know how to test and measure LLMs. Yes, some failure modes are known, but how many are not?

zer00eyz 4y ago
>. Tests of humans have evolved over a long time and large sample size, and humans may be more similar to each other than LLMs are, so failure modes may be more universal. In reading this the idea that sociopaths and psychopaths pass as "normal" springs to mind. Is what an LLM doing any different than what these people do? https://medium.datadriveninvestor.com/the-best-worst-funniest-most-absurd-etc-chatgpt-responses-9094dda976fb https://medium.datadriveninvestor.com/the-best-worst-funnies... For people language is spoken before it is written... there is a lot of biology in the spoken word (visual and audio queue)... I think without these these sorts of models are going to hit a wall pretty quickly.
- brookst 4y ago
  > In reading this the idea that sociopaths and psychopaths pass as "normal" springs to mind. > Is what an LLM doing any different than what these people do? I think it's too big of a question to have any meaning. Which sociopaths? Which LLMs? For what differences? It's like asking "is a car any different from an airplane"? Yes, obviously in some ways. No, they are identical in other ways.