2 ms·
> I may not trust ChatGPT's medical diagnosis or legal analysis, but it does pass those exams; It's really, really, really important to note that this could ea
by disgruntledphd2 2y ago
> I may not trust ChatGPT's medical diagnosis or legal analysis, but it does pass those exams;
It's really, really, really important to note that this could easily just be memorisation (and indeed, the GSM Symbolic paper from Apple suggests that it most likely is).
We don't have a good way of assessing the ability of models literally trained on all free text in existence, but comparing their performance based on text they've seen in training is definitely not it.
- ben_w 2y agoWhat I've always disliked about calling it "memorisation" is that we already have machines with much better memorisation than what LLMs do: the hard drives with the source documents. To the extent that it's able to respond to and with natural language, this is already an improvement over, say, grep or PageRank. To the extent that it is memorisation in the way we mean when we say a human has memorised something, it's an example of another aspect of what was meant by "intelligence" when I was growing up. It's definitely weird and alien as an intelligence, but it still ticks the boxes we used to have even a decade ago, 2014, let alone the late 80s to 1999 when I turned 16.