4 ms·

I asked a few LLMs whether zero is an even or odd number to see how they fared on such a widely misunderstood question: LLAMA 2 7B: 3x even, 2x neither LLAMA

I asked a few LLMs whether zero is an even or odd number to see how they fared on such a widely misunderstood question: LLAMA 2 7B: 3x even, 2x neither LLAMA 2 13B: 1x even, 4x neither LLAMA 2 70B: 1x even, 4x neither GPT 3.5: 20x even despite trying to trick it Mistral 7B: 1x even, 3x neither, 1x even+neither?! Mixtral 8x7B: 8x even, 2x neither

Klaus23 3y ago
It seems that it is not a matter of size or processing power, but a fundamental problem of bad training data that bakes these misconceptions into the model. OpenAI is probably using a better filtered dataset and may even be able to remove some of the more common errors through alignment and fine-tuning. I experimented a bit more and it seems that Mixtral is better than Mistral when it comes to common misconceptions. There has probably been some improvement in the training data.