3 ms·
From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not
by dsrtslnd23 3y ago
From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.
- janekm 3y agoThe evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjPMZb4fAED4qRHb?usp=sharing https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving. Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")