The 2022 survey is interesting - he marks "Write python code to implement algos like quicksort" and "Answer easily googleable factual but open-ended questions b

The 2022 survey is interesting - he marks "Write python code to implement algos like quicksort" and "Answer easily googleable factual but open-ended questions better than an expert" as "already happened" and the experts estimate 3 and 5 years in the future - this points to his expectations for what "happened" different than the experts. Like for the former "implement quicksort" they might be answering "write quicksort given an understanding of python and an understanding of the problem" rather than what happened which was "implement quicksort by reconstructing text from corpus". And on the latter - I'm not sure that we've achieved the "answer easily googleable factual questions better than an expert", unless by better he means answer it with more confidence than an expert (sarcasm). LLMs will flat out make up facts for factual questions and answer with complete confidence and there is no easy way to extract structured data out of those answers to confirm them. I think the idea of the marking these items as "these happened" is very handwavey and removes some of the interesting things about the survey. The most interesting thing in this article is where he compares the two articles and tracks deltas between the time estimates - I think that's a strong indicator of what we're finding is easier than we though in AI vs what is proving to be a bigger problem than we though in AI.