3 ms·

I don't think an exam suitable to asses a human is also suitable to asses a computer program, at least not any more. The only way a human can pass an exam like

I don't think an exam suitable to asses a human is also suitable to asses a computer program, at least not any more. The only way a human can pass an exam like that is to have studies countless hours and to have the required experience. The exam is not the same as the required expertise, but it is a reliable enough proxy. Also for humans there have been bullshits artists who could pass it. The scene from Catch me if you can is based on a true story. But for computers bullshitting is an artform. Passing the exam is no longer a good proxy for the required expertise. Another form of examination is now necessary for silicon applicants.

ben_w 4y ago
Makes sense. "When a measure becomes a target, it ceases to be a good measure," as the saying goes. This has already been applied to humans taking exams (studying for the test). My impression of chatGPT is like a super-genius with Alzheimer's: really impressive, but also not right.
epistemer 4y ago
It seems like there should be a control group of a non-medical person googling the questions. That would defeat the purpose of the career advancement though that this type of "research" is designed for. It is just a smart move by the authors, many citations incoming.