4 ms·
I agree 100%. Doing a worse job IS an error. It should be treated as such. Or at the very least make that behavior opt-in. The default should not be pretending
by mapontosevenths 4mo ago
I agree 100%. Doing a worse job IS an error. It should be treated as such. Or at the very least make that behavior opt-in. The default should not be pretending like nothing happened and just quietly doing a worse job.
Imagine your healthcare provider just sometimes decided not to read your test results very carefully and you risked death? Now realize that healthcare providers use Claude now and that scenario wasn't hypothetical.
- largbae 4mo agoEspecially if your name has any machine learning terms in it. Ah "Mr. Monty Carlo", it says here that you have a UTI, we'll get those kidneys removed ASAP so that won't happen again.
- ceejayoz 4mo agoYes, but as with spam/phishing/abuse prevention, too much information about what does and doesn't trigger things can be very useful to attackers. An explicit error is something you can feed into another AI to find jailbreaks. I think it's a fundamentally impossible thing to fix, though. There's no 100% correct answer.
- mapontosevenths 4mo agoI understand completely, and respect the tough spot they're in. They have a choice between human safety and cybersecurity/business needs here. I don't envy that position. That said, this thing is in real production use with war fighters, doctors, and financial experts. Just YOLO'ing to a dumber model midway through a multi-step process and pretending everything is fine is not a real or defensible option. Someone is going to die, and its going to be the fault of whoever decided to make this the default rather than opt-in. Personally, I couldn't live with myself.