An important thing to keep in mind when reading articles about GPT's test-taking abilities is that for humans, the tests are merely proxies. The point of taking

An important thing to keep in mind when reading articles about GPT's test-taking abilities is that for humans, the tests are merely proxies. The point of taking the bar exam isn't that we need people to be able to pass the bar exam, it's that (we hope) the only way a human can pass the bar is if they also have all the necessary skills to be a lawyer. The performance on the test is supposed to imply an additional suite of capabilities. But it's not clear that this should be true for GPT. In many cases, it may be possible for a language model to pass the test without having the capabilities the test is intended to establish. In this case, though, the test is just dumb, so the above is moot.