4 ms·
The author notes in their Twitter announcement [a] that their model’s reasoning abilities are only validated within the domain directly within their the domain
by nxobject 2y ago
The author notes in their Twitter announcement [a] that their model’s reasoning abilities are only validated within the domain directly within their the domain of their Countdown training material. They admit that the real test of this training method is whether it produces outputs that pass the sniff test in other subject domains, or even abstract reasoning. However, given that there are “standardized test style” abstract reasoning tests with relatively small corpora (eg. ZebraLogic [b] on the order of 1000 or so cases), I do think they missed an opportunity to… do _some_ small benchmark for abstract reasoning before announcement.
[a] https://threadreaderapp.com/thread/1882839370505621655.html https://threadreaderapp.com/thread/1882839370505621655.html - thanks @Tepix
[b] https://huggingface.co/blog/yuchenlin/zebra-logic https://huggingface.co/blog/yuchenlin/zebra-logic