4 ms·

someone can run tests and see that models output exactly the same results, and then you are open to criminal investigation.

clhodapp 3mo ago
Models don't even agree with themselves in terms of returning identical results
- tracerbulletx 3mo ago
  Yes they do. Sampling is the only pseudo-random part. Models return a deterministic distribution of output tokens for a given input of tokens.
wyrdcurt 3mo ago
Except that even the exact same model won't output the exact same results, that's a fundamental aspect of how LLMs work. They're probabilistic/stochastic, not deterministic.
- andriy_koval 3mo ago
  Models are weights for matrix operations, they are determenistics.
  - wyrdcurt 3mo ago
    They are weights for matrix operations, so in principle you'd think they would be deterministic. In practice it's more complicated than that. Not to be snarky or dismissive, I mean this genuinely: ask an LLM about it. I currently have a headache so I'm not up to explaining the technical details, but they are interesting and worth reading about.
  - antonvs 3mo ago
    Achieving determinism with LLMs and other neural network models is actually a hard problem that people spend a lot of time on, when they need that. It doesn’t happen by accident. Issues include accumulated floating point errors happening in different orders due to distributed and parallel computation, CUDA kernels that deliberately sacrifice determinism for speed, and several other such issues.
    - andriy_koval 3mo ago
      It happened that I am working on OSS LLM -> finetuning -> benchmark with 100k tests pipeline, and unless I do some data augmentation, result is 100% deterministic. I think you likely right, that some parts of stack could induce some marginal float point error, but converged model can mitigate it, and on some principal set of knowledge can give deterministic result with high probability. Which leads me to believe if you give this task to Anthropic, who has very strong incentive, they will build such benchmark, and then can tell that benchmark gives correct answer with 99.9% probability and it will be enough to drag someone to court.
      - nareshshah139 3mo ago
        The major cause for non-determism is batching.
      - antonvs 3mo ago
        If you're running on a single machine with a single GPU, then you may get deterministic results, although it still depends a lot on the details. For example if you're using Pytorch, you need to enable deterministic algorithms and may need to configure some other things as well. However, running in production at any sort of scale often involves multiple machines and multiple GPUs, and at that point, determinism can be difficult to achieve.
        andriy_koval 3mo ago
        No, I am not running on single GPU, but even then, nothing prevents to use single GPU for models verification.
  - monocasa 3mo ago
    The output is a probability distribution for all potential tokens. Then a "temperature" is applied to weight the sampling randomly (unless the temperature is zero in which case the stack can be deterministic and simply the highest probability token is selected). They go through this rigamarole because a little bit of randomness gives better results from a Turing test kind of perspective.
    - andriy_koval 3mo ago
      thank you, I already know basics.
xprnio 3mo ago
On something that is inherently non-deterministic? Something which is also to a great extent distilled from other frontier models, meaning it has the possibility to generate similar outputs to those meaning that just pattern detection might also not be as effective? Easier to ban everything that’s open, than try to figure out which one of them is Chinese
- andriy_koval 3mo ago
  Now imagine prosecutor found expert, who said there is benchmark which while performing 100k test questions found it is the same model with 98% probability, and then you need under oath testify where did you get this model.
  - m11a 3mo ago
    Step 1: Chinese company publishes open weights on HF Step 2: European company distills or just adjusts the model slightly, and publishes its model on HF Step 3: American company uses model from step 2. Has to testify under oath where they got it from. "We got it from these French guys"
    - andriy_koval 3mo ago
      That French guy takes risk to be forever under US warrants for breaking American law, denied access to financial institutions even in Europe and will quickly go to some KYC entity list, and you will be notified as his clients to stop using his model. Or you think all kind of fraud can be committed through some "french guy"? Also, I am not confident, receiving illegal materials from French guy gates you from personal liability.
      - m11a 3mo ago
        I presume such US legislation isn't going to try claim worldwide jurisdiction to block all persons worldwide from using Chinese models. In which case, the French guy wouldn't be violating American law. As for the American company, it's pretty difficult to check the provedance of open weights. It's even difficult to check the provedance of open source code, because chains of attribution aren't always clear. I posted elsewhere that Anthropic's MCP Python SDK is a fork of an open source project with the attribution removed. We saw the same with Cursor's Composer model, which didn't attribute its Chinese base. It's very hard to claim an American company should be liable for using a purportedly European model with attribution removed.
antonvs 3mo ago
Many of these models will report that they are Claude. It’s going to be difficult to overcome reasonable doubt.

clhodapp 3mo ago

Models don't even agree with themselves in terms of returning identical results

tracerbulletx 3mo ago
Yes they do. Sampling is the only pseudo-random part. Models return a deterministic distribution of output tokens for a given input of tokens.

wyrdcurt 3mo ago

Except that even the exact same model won't output the exact same results, that's a fundamental aspect of how LLMs work. They're probabilistic/stochastic, not deterministic.

andriy_koval 3mo ago
Models are weights for matrix operations, they are determenistics.
- wyrdcurt 3mo ago
  They are weights for matrix operations, so in principle you'd think they would be deterministic. In practice it's more complicated than that. Not to be snarky or dismissive, I mean this genuinely: ask an LLM about it. I currently have a headache so I'm not up to explaining the technical details, but they are interesting and worth reading about.
- antonvs 3mo ago
  Achieving determinism with LLMs and other neural network models is actually a hard problem that people spend a lot of time on, when they need that. It doesn’t happen by accident. Issues include accumulated floating point errors happening in different orders due to distributed and parallel computation, CUDA kernels that deliberately sacrifice determinism for speed, and several other such issues.
  - andriy_koval 3mo ago
    It happened that I am working on OSS LLM -> finetuning -> benchmark with 100k tests pipeline, and unless I do some data augmentation, result is 100% deterministic. I think you likely right, that some parts of stack could induce some marginal float point error, but converged model can mitigate it, and on some principal set of knowledge can give deterministic result with high probability. Which leads me to believe if you give this task to Anthropic, who has very strong incentive, they will build such benchmark, and then can tell that benchmark gives correct answer with 99.9% probability and it will be enough to drag someone to court.
    - nareshshah139 3mo ago
      The major cause for non-determism is batching.
    - antonvs 3mo ago
      If you're running on a single machine with a single GPU, then you may get deterministic results, although it still depends a lot on the details. For example if you're using Pytorch, you need to enable deterministic algorithms and may need to configure some other things as well. However, running in production at any sort of scale often involves multiple machines and multiple GPUs, and at that point, determinism can be difficult to achieve.
      - andriy_koval 3mo ago
        No, I am not running on single GPU, but even then, nothing prevents to use single GPU for models verification.
- monocasa 3mo ago
  The output is a probability distribution for all potential tokens. Then a "temperature" is applied to weight the sampling randomly (unless the temperature is zero in which case the stack can be deterministic and simply the highest probability token is selected). They go through this rigamarole because a little bit of randomness gives better results from a Turing test kind of perspective.
  - andriy_koval 3mo ago
    thank you, I already know basics.

xprnio 3mo ago

On something that is inherently non-deterministic? Something which is also to a great extent distilled from other frontier models, meaning it has the possibility to generate similar outputs to those meaning that just pattern detection might also not be as effective? Easier to ban everything that’s open, than try to figure out which one of them is Chinese

andriy_koval 3mo ago
Now imagine prosecutor found expert, who said there is benchmark which while performing 100k test questions found it is the same model with 98% probability, and then you need under oath testify where did you get this model.
- m11a 3mo ago
  Step 1: Chinese company publishes open weights on HF Step 2: European company distills or just adjusts the model slightly, and publishes its model on HF Step 3: American company uses model from step 2. Has to testify under oath where they got it from. "We got it from these French guys"
  - andriy_koval 3mo ago
    That French guy takes risk to be forever under US warrants for breaking American law, denied access to financial institutions even in Europe and will quickly go to some KYC entity list, and you will be notified as his clients to stop using his model. Or you think all kind of fraud can be committed through some "french guy"? Also, I am not confident, receiving illegal materials from French guy gates you from personal liability.
    - m11a 3mo ago
      I presume such US legislation isn't going to try claim worldwide jurisdiction to block all persons worldwide from using Chinese models. In which case, the French guy wouldn't be violating American law. As for the American company, it's pretty difficult to check the provedance of open weights. It's even difficult to check the provedance of open source code, because chains of attribution aren't always clear. I posted elsewhere that Anthropic's MCP Python SDK is a fork of an open source project with the attribution removed. We saw the same with Cursor's Composer model, which didn't attribute its Chinese base. It's very hard to claim an American company should be liable for using a purportedly European model with attribution removed.

antonvs 3mo ago

Many of these models will report that they are Claude. It’s going to be difficult to overcome reasonable doubt.