3 ms·
I checked the original research by BBC but the underlying data was not published - no way to replicate the study, and no way to check its validity. They didn’t
by kolinko 2y ago
I checked the original research by BBC but the underlying data was not published - no way to replicate the study, and no way to check its validity.
They didn’t provide the answers models provided, so we can’t even say if the reviews of the answers were correct. And since the experiment relied on BBC giving timed access to their archive to the models, we couldn’t even type in the same prompts into the models to see the model outputs.
- newsymonger 2y agoI think what little we do know about the research also exposes flaws. For example, although the BBC prompted the AI to only use BBC sources, it admits that other sources were used to compile responses. But the BBC only judged the accuracy of the answers against its own reporting. So, the AI summaries might well have contained info that wasn't in the BBC reports, but it may have gleaned those from other sources. It would be interesting to see the experiment repeated, but with the BBC feeding the AI PDFs/text of the initial reports. I suspect it would be much more accurate.
- 72deluxe 2y agoWitty isn't it. It seems to have all the hallmarks of the same "total cost of ownership" study of Linux that Microsoft issued during Ballmer's years...
- amai 2y agoHave a look at the appendix of https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-into-ai-assistants.pdf https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-int... There you will find at least some examples of questions and answers. But it is true, the whole raw dataset has not been published, yet.