5 ms·

As always, I tried my favorite prompt: > I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The quali

As always, I tried my favorite prompt: > I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap. Claude 2 got really close to the right answer on the first try. The only thing it didn't consider is that time for 9.6 laps means driving 10 laps total, since incomplete laps must be finished after time is up. For the record, GPT-4 does take even this catch into account and gives a perfect answer, while Bard outputs results that are off by orders of magnitude. Claude 2's response was: > Okay, let's calculate this: > * Your qualifying time of 2:04.317 equates to an average lap time of 124.317 seconds > * A 20 minute race = 20 * 60 = 1200 seconds > * So in a 20 minute race you can do roughly 1200 / 124.317 = 9.67 laps > * At 2.73 liters per lap, that's 9.67 * 2.73 = 26.38 liters of fuel needed for the race > So I would recommend taking on at least 27-28 liters of fuel to be safe for a 20 minute race, given your average lap times and fuel consumption. It's always better to take on a bit more fuel in case there are any safety cars or variations in your lap times during the race. Let me know if you have any other questions!

moffkalast 3y ago
Forgot the final one liter required for the FIA sample, gottem.
GGO 3y ago
Google should train bard on their own interview questions. Maybe it will get this right after that
Lerc 3y ago
When you test this prompt, do you do multiple separate queries to see how much variance there is in the answers?
majormajor 3y ago
> The only thing it didn't consider is that time for 9.6 laps means driving 10 laps total, since incomplete laps must be finished after time is up. I would be hesitant to judge an LLM on the various ground rules of the real world that it happens to have internalized or not, that aren't given in the prompt. For my uses I'd want to have prompts that provide necessary context anyway - and would actually favor ones that can do it from the prompt without needing to already know - so that it is not limited when it encounters a novel problem domain.
Therenas 3y ago
Interesting that is mentions the safety car making you need more fuel, when really you need less because of the reduced speed under it.
sameline 3y ago
Eventually one of these comment threads is going to be included in the training set invalidating this as a test.
- shri_krishna 3y ago
  Which is why knowledge cut off date is important. I prefer if it is frozen to pre-ChatGPT-3.5. Anything post-ChatGPT-3.5 release date should be considered tainted - imagine the sheer number of articles generated by spammers who used ChatGPT.
  - gridspy 3y ago
    That might be fair in the short term. However it's not a workable option long-term, or all such models will be very limited in their knowledge as humanity advances technologically and culturally.
    - shri_krishna 3y ago
      If you want me to be honest with you, LLMs are themselves a short term approach and can get us to, at max, AGI levels (for this current era). I don't see us getting to ASI with just LLMs. For the sort of "emergent ability" that ASI requires it has to be something more "simpler" and the learning be more "virulent" / "instantaneous" (not sure if these words convey what I really want to convey). Otherwise, LLMs will always have a "maxima" at which point it fails. And that maxima is collective intelligence of all of humanity in the current epoch. If you go back a 1000 years, the collective intelligence of all humanity would be completely different (primitive even). Would LLMs trained on that data have produced Knowledge that we know today? I don't think so. It could still, theoretically, reach AGI for that era and accelerate pace of learning by 50-100 years at a time. LLMs will surely accelerate pace of learning (as tools) even now but by themselves won't reach ASI levels. For ASI, we really need something more simpler/fundamental that is yet to be discovered. I don't feel LLMs are the way to ASI. AGI? Yeah possible.
      - visarga 3y ago

moffkalast 3y ago

Forgot the final one liter required for the FIA sample, gottem.

GGO 3y ago

Google should train bard on their own interview questions. Maybe it will get this right after that

Lerc 3y ago

When you test this prompt, do you do multiple separate queries to see how much variance there is in the answers?

majormajor 3y ago

> The only thing it didn't consider is that time for 9.6 laps means driving 10 laps total, since incomplete laps must be finished after time is up. I would be hesitant to judge an LLM on the various ground rules of the real world that it happens to have internalized or not, that aren't given in the prompt. For my uses I'd want to have prompts that provide necessary context anyway - and would actually favor ones that can do it from the prompt without needing to already know - so that it is not limited when it encounters a novel problem domain.

Therenas 3y ago

Interesting that is mentions the safety car making you need more fuel, when really you need less because of the reduced speed under it.

sameline 3y ago

Eventually one of these comment threads is going to be included in the training set invalidating this as a test.

shri_krishna 3y ago
Which is why knowledge cut off date is important. I prefer if it is frozen to pre-ChatGPT-3.5. Anything post-ChatGPT-3.5 release date should be considered tainted - imagine the sheer number of articles generated by spammers who used ChatGPT.
- gridspy 3y ago
  That might be fair in the short term. However it's not a workable option long-term, or all such models will be very limited in their knowledge as humanity advances technologically and culturally.
  - shri_krishna 3y ago
    If you want me to be honest with you, LLMs are themselves a short term approach and can get us to, at max, AGI levels (for this current era). I don't see us getting to ASI with just LLMs. For the sort of "emergent ability" that ASI requires it has to be something more "simpler" and the learning be more "virulent" / "instantaneous" (not sure if these words convey what I really want to convey). Otherwise, LLMs will always have a "maxima" at which point it fails. And that maxima is collective intelligence of all of humanity in the current epoch. If you go back a 1000 years, the collective intelligence of all humanity would be completely different (primitive even). Would LLMs trained on that data have produced Knowledge that we know today? I don't think so. It could still, theoretically, reach AGI for that era and accelerate pace of learning by 50-100 years at a time. LLMs will surely accelerate pace of learning (as tools) even now but by themselves won't reach ASI levels. For ASI, we really need something more simpler/fundamental that is yet to be discovered. I don't feel LLMs are the way to ASI. AGI? Yeah possible.
    - visarga 3y ago