3 ms·

LLMs produce deterministic results? Now, that's a big [citation needed]. Where can I find the specs? Edit: This is assuming by "deterministic," you mean the sa

LLMs produce deterministic results? Now, that's a big [citation needed]. Where can I find the specs? Edit: This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points.

soulofmischief 2y ago
I'd recommend learning how transformers work, and the concept of temperature. I don't think I need to cite information that is broadly and readily available, but here: https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... I also qualified the requirement of needing the same hardware, due to FP shenanigans. I could further clarify that you need the same stack (pytorch, tensorflow, etc)
- soraminazuki 2y ago
  This gcc script that I created below is just as "deterministic" as an LLM. It produces the same result every time. Doesn't make it useful though. echo '#!/usr/bin/env bash' > gcc echo 'cat <<EOF' >> gcc openssl rand -base64 100 >> gcc echo 'EOF' >> gcc chmod +x gcc Also, how transformers work is not a spec of the LLM that anyone can use to learn how LLM produces code. It's no gcc source code.
  - soulofmischief 2y ago
    You claimed they weren't deterministic, I have shown that they can be. I'm not sure what your point is. And it is incorrect to base your analysis of future transformer performance on current transformer performance. There is a lot of ongoing research in this area and we have seen continual progress.
    - soraminazuki 2y ago
      I reiterate: > This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points. "Determinism" is a word that you brought up in response to my comment, which I charitably interpreted to mean the same thing I was originally talking about. Also, it's 100% correct to analyze things based on its fundamental properties. It's absurd to criticize people for assuming 2 + 2 = 4 because "continual progress" might make it 5 in the future.
      - soulofmischief 2y ago
        What are these fundamental properties you speak of? 8 years ago this was all a pipe dream. Are you claiming to know what the next 8 years of transformer development will look like?
        soraminazuki 2y ago
        That LLMs are by definition models of human speech and have no cognitive capabilities. There is no sound logic behind what LLMs spit out, and will stay that way because it merely mimics its training data. No amount of vague future transformers will transform away how the underlying technology works. But let's say we have something more than an LLM, that still wouldn't make natural languages a good replacement for programming languages. This is because natural languages are, as the article mentions, imprecise. It just isn't a good tool. And no, transformers can't change how languages work. It can only "recontextualize," or as some people might call it, "hallucinate."
        soulofmischief 2y ago
        Citation needed. Modern transformers are much, much more than just speech models. Precisely define "cognitive capabilities", and provide proof as to why neural models cannot ever mimic these cognitive capabilities. > But let's say we have something more than an LLM We do. Modern multi-modal transformers. > This is because natural languages are, as the article mentions, imprecise Two different programmers can take a well-enough defined spec and produce two separate code bases that may (but not must) differ in implementation, while still having the exact same interfaces and testable behavior. > And no, transformers can't change how languages work. It can only "recontextualize," or as some people might call it, "hallucinate." You don't understand recontextualization if you think it means hallucination. Or vice versa. Hallucination is about returning incorrect or false data. Recontextualization is akin to decompression, and can be lossy or "effectively" lossless (within a probabilistic framework; again, the interfaces and behavior just need to match)

soulofmischief 2y ago

I'd recommend learning how transformers work, and the concept of temperature. I don't think I need to cite information that is broadly and readily available, but here: https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... I also qualified the requirement of needing the same hardware, due to FP shenanigans. I could further clarify that you need the same stack (pytorch, tensorflow, etc)

soraminazuki 2y ago
This gcc script that I created below is just as "deterministic" as an LLM. It produces the same result every time. Doesn't make it useful though. echo '#!/usr/bin/env bash' > gcc echo 'cat <<EOF' >> gcc openssl rand -base64 100 >> gcc echo 'EOF' >> gcc chmod +x gcc Also, how transformers work is not a spec of the LLM that anyone can use to learn how LLM produces code. It's no gcc source code.
- soulofmischief 2y ago
  You claimed they weren't deterministic, I have shown that they can be. I'm not sure what your point is. And it is incorrect to base your analysis of future transformer performance on current transformer performance. There is a lot of ongoing research in this area and we have seen continual progress.
  - soraminazuki 2y ago
    I reiterate: > This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points. "Determinism" is a word that you brought up in response to my comment, which I charitably interpreted to mean the same thing I was originally talking about. Also, it's 100% correct to analyze things based on its fundamental properties. It's absurd to criticize people for assuming 2 + 2 = 4 because "continual progress" might make it 5 in the future.
    - soulofmischief 2y ago
      What are these fundamental properties you speak of? 8 years ago this was all a pipe dream. Are you claiming to know what the next 8 years of transformer development will look like?
      - soraminazuki 2y ago
        That LLMs are by definition models of human speech and have no cognitive capabilities. There is no sound logic behind what LLMs spit out, and will stay that way because it merely mimics its training data. No amount of vague future transformers will transform away how the underlying technology works. But let's say we have something more than an LLM, that still wouldn't make natural languages a good replacement for programming languages. This is because natural languages are, as the article mentions, imprecise. It just isn't a good tool. And no, transformers can't change how languages work. It can only "recontextualize," or as some people might call it, "hallucinate."
        soulofmischief 2y ago
        Citation needed. Modern transformers are much, much more than just speech models. Precisely define "cognitive capabilities", and provide proof as to why neural models cannot ever mimic these cognitive capabilities. > But let's say we have something more than an LLM We do. Modern multi-modal transformers. > This is because natural languages are, as the article mentions, imprecise Two different programmers can take a well-enough defined spec and produce two separate code bases that may (but not must) differ in implementation, while still having the exact same interfaces and testable behavior. > And no, transformers can't change how languages work. It can only "recontextualize," or as some people might call it, "hallucinate." You don't understand recontextualization if you think it means hallucination. Or vice versa. Hallucination is about returning incorrect or false data. Recontextualization is akin to decompression, and can be lossy or "effectively" lossless (within a probabilistic framework; again, the interfaces and behavior just need to match)