4 ms·

I think determinism is an important element. You can ask the same LLM the same question repeatedly and get different answers - and not just different ways of st

I think determinism is an important element. You can ask the same LLM the same question repeatedly and get different answers - and not just different ways of stating the same answer, very different answers. If you ask an intelligent being the same question they may occasionally change the precise words they use but their answer will be the same over and over.

samuel 2y ago
That's not an inherent property of the system. You can choose the most likely token(topk=1) and it will be deterministic (at least in theory, in some hardware setups it might be trickier)
hdjjhhvvhga 2y ago
If determinism is a hard requirement, then LLM-based AI can't fulfill it by definition.
beering 2y ago
A human will give different answers to the same question, so I’m not sure why it’s fair to set a higher bar for an LLM. Or rather, I’m not sure how you would design this test in a way where humans would pass and the best LLM would fail.
simonw 2y ago
That's because "intelligent beings" have memory. If you ask an LLM the same question within the same chat session you'll get a consistent answer about it.
- Arainach 2y ago
  I disagree. If you were to take a snapshot of someone's knowledge and memory such that you could restore to it over and over, that person would give the same answer to the question. The same is not true for an LLM. Heck, I can't even get LLMs to be consistent about *their own capabilities*. Bias disclaimer: I work at Google, but not on Gemini. If I ask Gemini to produce an SVG file, it will sometimes do so and sometimes say "sorry, I can't, I can only produce raster images". I cannot deterministically produce either behavior - it truly seems to vary randomly.
  - IanCal 2y ago
    You could run an llm deterministically too. We're often explicitly adding in randomness to the results so it feels weird to then accuse them of not being intelligent after we deliberately force them off the path.
  - danielbln 2y ago
    You'd need to restore more than memory/knowledge. You'd need to restore the full human, and in the exact same condition (inside and out). Ask me some question before bed and again after waking up, I'll probably answer it at night but in the morning tell you to sod off until I had coffee.
  - int_19h 2y ago
    Of course it varies randomly, that's literally what temperature is for in generation.

samuel 2y ago

That's not an inherent property of the system. You can choose the most likely token(topk=1) and it will be deterministic (at least in theory, in some hardware setups it might be trickier)

hdjjhhvvhga 2y ago

If determinism is a hard requirement, then LLM-based AI can't fulfill it by definition.

beering 2y ago

A human will give different answers to the same question, so I’m not sure why it’s fair to set a higher bar for an LLM. Or rather, I’m not sure how you would design this test in a way where humans would pass and the best LLM would fail.

simonw 2y ago

That's because "intelligent beings" have memory. If you ask an LLM the same question within the same chat session you'll get a consistent answer about it.

Arainach 2y ago
I disagree. If you were to take a snapshot of someone's knowledge and memory such that you could restore to it over and over, that person would give the same answer to the question. The same is not true for an LLM. Heck, I can't even get LLMs to be consistent about *their own capabilities*. Bias disclaimer: I work at Google, but not on Gemini. If I ask Gemini to produce an SVG file, it will sometimes do so and sometimes say "sorry, I can't, I can only produce raster images". I cannot deterministically produce either behavior - it truly seems to vary randomly.
- IanCal 2y ago
  You could run an llm deterministically too. We're often explicitly adding in randomness to the results so it feels weird to then accuse them of not being intelligent after we deliberately force them off the path.
- danielbln 2y ago
  You'd need to restore more than memory/knowledge. You'd need to restore the full human, and in the exact same condition (inside and out). Ask me some question before bed and again after waking up, I'll probably answer it at night but in the morning tell you to sod off until I had coffee.
- int_19h 2y ago
  Of course it varies randomly, that's literally what temperature is for in generation.