3 ms·

I tried my standard testprompt: Two cars have a 100 mile race. Car A drives 10 miles per hour. Car B drives 5 miles per hour, but gets a 10 hour he

by mg 2y ago

I tried my standard testprompt: Two cars have a 100 mile race. Car A drives 10 miles per hour. Car B drives 5 miles per hour, but gets a 10 hour headstart. Who wins? And the Liquid-40B model failed with a long explanation why car B wins the race. Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a headstart. Because every time the fast runner gets to the point where he saw the slow runner, the slow runner is already a bit further along the track.

leetharris 2y ago
To be fair, this article does say: >What are Language LFMs not good at today: >Zero-shot code tasks >Precise numerical calculations >Time-sensitive information >Counting r's in the word "Strawberry"! >Human preference optimization techniques have not been applied extensively to our models yet. Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.
- aithrowawaycomm 2y ago
  But they also say > What are Language LFMs good at today: > General and expert knowledge > Mathematics and logical reasoning "Our LLM is good at mathematics but bad at computation" doesn't really cut the mustard here. What they mean is "good at well-known math benchmarks but bad at simple math problems that it hasn't been trained on." The idea that this constitutes "mathematics and logical reasoning" is a testament to AI companies' poor scientific standards.
vletal 2y ago
Thanks! I am collecting all "test prompts" which appear at HN and Reddit. I plan to create a hugging face dataset. I will soon publish "Vojta-1B", which is gonna pass all of them.
wazokazi 2y ago
I got this >> To determine the winner, we need to calculate the time it takes for each car to complete the 100-mile race. For Car A: Speed = 10 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 10 miles per hour = 10 hours For Car B: Speed = 5 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 5 miles per hour = 20 hours However, Car B has a 10-hour headstart, so we need to subtract that from its total time: Adjusted Time for Car B = 20 hours - 10 hours = 10 hours Both cars take 10 hours to complete the race, so they tie.
- reustle 2y ago
  What about acceleration times? :-) 0-5 vs 0-10 should be slightly different

by mg 2y ago

leetharris 2y ago

To be fair, this article does say: >What are Language LFMs not good at today: >Zero-shot code tasks >Precise numerical calculations >Time-sensitive information >Counting r's in the word "Strawberry"! >Human preference optimization techniques have not been applied extensively to our models yet. Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.

aithrowawaycomm 2y ago
But they also say > What are Language LFMs good at today: > General and expert knowledge > Mathematics and logical reasoning "Our LLM is good at mathematics but bad at computation" doesn't really cut the mustard here. What they mean is "good at well-known math benchmarks but bad at simple math problems that it hasn't been trained on." The idea that this constitutes "mathematics and logical reasoning" is a testament to AI companies' poor scientific standards.

vletal 2y ago

Thanks! I am collecting all "test prompts" which appear at HN and Reddit. I plan to create a hugging face dataset. I will soon publish "Vojta-1B", which is gonna pass all of them.

wazokazi 2y ago

I got this >> To determine the winner, we need to calculate the time it takes for each car to complete the 100-mile race. For Car A: Speed = 10 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 10 miles per hour = 10 hours For Car B: Speed = 5 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 5 miles per hour = 20 hours However, Car B has a 10-hour headstart, so we need to subtract that from its total time: Adjusted Time for Car B = 20 hours - 10 hours = 10 hours Both cars take 10 hours to complete the race, so they tie.

reustle 2y ago
What about acceleration times? :-) 0-5 vs 0-10 should be slightly different