You should be more skeptical of their benchmarks as the people choosing the tests are also the ones trying to show their system passes. Their thumb is totally o

You should be more skeptical of their benchmarks as the people choosing the tests are also the ones trying to show their system passes. Their thumb is totally on the scale. Just look at the press release for GPT4, namely the graph showing where its improved most relative to 3.5. The largest margin of improvement was the uniform bar exam. Do you think that is a coincidence that happens by just agnostically feeding more data and doing more training without favoring anything in particular? Of course not. Law is a potential multi billion dollar annual revenue stream. The largest power of 2 less than 1031 is a multi dozen dollar annual revenue stream. Screw the goal of AGI, AI lawyer that can kinda do non-law stuff is what they can attain today! Any exam you can study for is a bad test of its ability to reason through novel situations. AP tests are not a great test for this purpose. Oh come on, how can a vision model be a competitor for a language model. That's a very tenuous leap of logic. I'll stop short of calling it motivated reasoning, but its certainly biased reasoning bordering on rationalizing.