3 ms·

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I de

by gurjeet 27d ago

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design. TERMy (or is it the NPC-forge) seems to be worth a try.

kouteiheika 27d ago
> because their output is non-deterministic by design. It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.
- kennywinker 27d ago
  That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response to very similar prompts. This tool has a finite amount of outputs for an infinite amount of inputs. Which is different from an llm based tool.
  - skeledrew 27d ago
    I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.
    - kennywinker 27d ago
      Yes i think I acknowledged that, but is that useful for making a tool that can be trusted to safely run shell commands when asked arbitrary questions? No. It’s not.
piterrro 27d ago
You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.
- jdiff 27d ago
  That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.
  - asQuirreL 27d ago
    That's the definition of a chaotic system (small change in initial conditions results in large, seemingly -- but not actually -- random changes in output), but it's still deterministic (same input results in same output).
  - kzrdude 27d ago
    I've long observed that kind of behaviour in google translate (which makes sense, they have been using ML for a long time.)

by gurjeet 27d ago

kouteiheika 27d ago

> because their output is non-deterministic by design. It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

kennywinker 27d ago
That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response to very similar prompts. This tool has a finite amount of outputs for an infinite amount of inputs. Which is different from an llm based tool.
- skeledrew 27d ago
  I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.
  - kennywinker 27d ago
    Yes i think I acknowledged that, but is that useful for making a tool that can be trusted to safely run shell commands when asked arbitrary questions? No. It’s not.

piterrro 27d ago

You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.

jdiff 27d ago
That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.
- asQuirreL 27d ago
  That's the definition of a chaotic system (small change in initial conditions results in large, seemingly -- but not actually -- random changes in output), but it's still deterministic (same input results in same output).
- kzrdude 27d ago
  I've long observed that kind of behaviour in google translate (which makes sense, they have been using ML for a long time.)