3 ms·
> Every commit is reproducible. The prompt is preserved with > some extra attributes about which model and agent was used. > You can re-run any commit against
by scheme271 7mo ago
> Every commit is reproducible. The prompt is preserved with
> some extra attributes about which model and agent was used.
> You can re-run any commit against a fresh checkout to see
> what Claude generates from the same instruction.
I don't see how this is true. LLMs can generate different outputs even with the same model and inputs.
- croisillon 7mo agoas do most humans
- lefrenchy 7mo agoAnd? The claim is about AI.
- fragmede 7mo agoHow different? So do compilers.
- Levitating 7mo agoCompilers of the same version should preferably not produce different output with the same input.
- fragmede 7mo ago"preferably" doesn't mean t that's the ground truth. Compilers aren't absolutely deterministic, only sufficiently so for most programmers to use.
- scheme271 7mo agoIf you have a compiler the same source code and the same options, it should generate the same output everything provided you aren't using some compiler pragmas or something similar that embeds timestamps or random numbers or similar. If you give an LLM the same input, it can generate different outputs (controlled by the temperature setting).
- fragmede 7mo agoSo provided you don't do anything that might cause it to be non-determinist, it's deterministic. Got it.
- scheme271 7mo agoI'll be charitable here but you need to go out of your way to introduce non-determinism. Bit reproducible builds and distros exist so it is possible to have an entire distro that can be reliably reproduced bit-by-bit on different systems and at different times.
- fragmede 7mo agoIt's the other direction. You have to put in an extreme amount of effort, like Debian has, which you can just piggyback off of, to cause determinism to be introduced. 2013 they started that initiative. They're reasonably there, thirteen years later, but to disregard the amount of effort it took to get there would be to forget history. Give ChatGPT thirteen more years to iterate, and see where it is then.
- ares623 7mo agoFolks who bring up these "gotchas" should be forbidden from using or taking advantage of the things they are disingenuously whataboutism-ing. Reminds me of sovereign citizen behavior.
- ares623 7mo agothe models themselves don't even last a few years. Do they even last a few weeks? Who knows how much they subtly change behind the scenes.