4 ms·
Also, MPT 7B gets it right over half the time. I've been testing every new LLM with that question. Also, I tend to include mention in the question that all sib
by jakderrida 3y ago
Also, MPT 7B gets it right over half the time. I've been testing every new LLM with that question.
Also, I tend to include mention in the question that all siblings are from the same two parents to preclude half-siblings because half my friends have half-siblings from both sides scattered across the country; so the wrong answers actually do tend to apply to them sometimes.
- panarky 3y ago> I've been testing every new LLM with that question We should pay more attention to data contamination when using popular prompts for testing.
- jakderrida 3y agoNo doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data for most LLMs yet.
- pixl97 3y agoIt could very well be in the data, but not captured via learning/encoding.
- jakderrida 3y agoThat's actually a really great point. I'm guessing we need to keep modifying attributes of the questions while maintaining the underlying structure. Instead of "Sally (a girl)", it's "Sal (a guy)" and then tweak the numbers. Although, part of me is convinced it's almost a fluke that MPT 7B gets it right because MPT 30B doesn't.