3 ms·
No doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data f
by jakderrida 3y ago
No doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data for most LLMs yet.
- pixl97 3y agoIt could very well be in the data, but not captured via learning/encoding.
- jakderrida 3y agoThat's actually a really great point. I'm guessing we need to keep modifying attributes of the questions while maintaining the underlying structure. Instead of "Sally (a girl)", it's "Sal (a guy)" and then tweak the numbers. Although, part of me is convinced it's almost a fluke that MPT 7B gets it right because MPT 30B doesn't.