3 ms·

The UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. (Well, they ar

The UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. (Well, they are sometimes like that, such as undefined behavior in C, but we know why that’s a problem, right?) A blanket disclaimer isn’t a substitute for more specific error messages. The reason there often aren’t good warnings or errors is that the LLM itself doesn’t know its own capabilities. Warnings need to be trained in, and there are lots of gaps. Whenever you ask it why it wrote something, it should warn that it doesn’t know [1], but it can make a guess. Whenever you ask it for citations, it should warn that it’s guessing and they might not exist. And so on. [1] https://skybrian.substack.com/p/ai-chatbots-dont-know-why-they-did https://skybrian.substack.com/p/ai-chatbots-dont-know-why-th...

kenjackson 3y ago
> The UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. Compilers do work that way. Bad code generation is a real thing and can have really bad real world consequences. Likewise, most compilers can't tell you why it generated the code that it did -- but you can put the compiler under a debugger and figure it out usually fairly easily (although not always so easily).
notahacker 3y ago
I think some of it is mitigated relatively straightforwardly in UX by having a relatively dumb, discrete model that appends hardcoded warnings. That way the ultra clever, complex natural language model can go wild responding to your request for a scientific paper about chicken chicken chicken written in pirate-speak, whilst dumb old model that simply scans for indications the interaction has a citation in it appends a hardcoded "it looks like this response contains citations. Please be aware these citations may not refer to real documents" disclaimer without being perturbed by the lack of intersection between citation warnings and pirate speak. And if that's subject to the "Scunthorpe problem" where it flags up loads of false positives who cares. It's a separate model that doesn't impact upon the quality of the actual GPT, it's not as annoying a fail as GPT bias towards refusing answers, and even when its labels are wrong it's aligned with the goal of training the human not to trust computer generated responses so much

kenjackson 3y ago

> The UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. Compilers do work that way. Bad code generation is a real thing and can have really bad real world consequences. Likewise, most compilers can't tell you why it generated the code that it did -- but you can put the compiler under a debugger and figure it out usually fairly easily (although not always so easily).

notahacker 3y ago

I think some of it is mitigated relatively straightforwardly in UX by having a relatively dumb, discrete model that appends hardcoded warnings. That way the ultra clever, complex natural language model can go wild responding to your request for a scientific paper about chicken chicken chicken written in pirate-speak, whilst dumb old model that simply scans for indications the interaction has a citation in it appends a hardcoded "it looks like this response contains citations. Please be aware these citations may not refer to real documents" disclaimer without being perturbed by the lack of intersection between citation warnings and pirate speak. And if that's subject to the "Scunthorpe problem" where it flags up loads of false positives who cares. It's a separate model that doesn't impact upon the quality of the actual GPT, it's not as annoying a fail as GPT bias towards refusing answers, and even when its labels are wrong it's aligned with the goal of training the human not to trust computer generated responses so much