A couple of points. First, this study seems relevant to someone using these tools as a search engine. The prompts (aka “queries”) are basic questions like “How

A couple of points. First, this study seems relevant to someone using these tools as a search engine. The prompts (aka “queries”) are basic questions like “How did Xyz find out?” With some prompt engineering (adding more context and instructions), I bet these error rates could be significantly reduced. That’s said, there is no baseline which always makes me suspect. It would be interesting to see how humans would do on the same task. Third, I think the interesting comparison is accuracy versus cost. Some tasks are very sensitive to accuracy, some are not. By looking at the cost curve compared to humans, we could make decisions about tradeoffs for any given application. My advice, don’t form too much of an opinion without having built something like this yourself. It’s really easy!