4 ms·

>sufficiently descriptive natural language specification https://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/ https://www.commitstri

>sufficiently descriptive natural language specification https://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/ https://www.commitstrip.com/en/2016/08/25/a-very-comprehensi...

soulofmischief 2y ago
No, the key difference is that an engineer becomes more product-oriented, and the technicalities of the implementation are deprioritized. It is a different paradigm, in the same way that a high-level language like JavaScript handles a lot of low-level stuff for me.
- soraminazuki 2y ago
  A programming language implementation produces results that are controllable, reproducible, and well-defined. An LLM has none of those properties, which makes the comparison moot. Having an LLM make up underspecified details willy-nilly, or worse, ignore clear instructions is very different from programming languages "handling a lot of low-level stuff."
  - soulofmischief 2y ago
    [citation needed] You can set temperature to 0 in many LLMs and get deterministic results (on the same hardware, given floating-point shenanigans). You can provide a well-defined spec and test suite. You can constrain and control the output.
    - soraminazuki 2y ago
      LLMs produce deterministic results? Now, that's a big [citation needed]. Where can I find the specs? Edit: This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points.
      - soulofmischief 2y ago
        I'd recommend learning how transformers work, and the concept of temperature. I don't think I need to cite information that is broadly and readily available, but here: https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... I also qualified the requirement of needing the same hardware, due to FP shenanigans. I could further clarify that you need the same stack (pytorch, tensorflow, etc)
        soraminazuki 2y ago
        This gcc script that I created below is just as "deterministic" as an LLM. It produces the same result every time. Doesn't make it useful though. echo '#!/usr/bin/env bash' > gcc echo 'cat <<EOF' >> gcc openssl rand -base64 100 >> gcc echo 'EOF' >> gcc chmod +x gcc Also, how transformers work is not a spec of the LLM that anyone can use to learn how LLM produces code. It's no gcc source code.
        soulofmischief 2y ago
        You claimed they weren't deterministic, I have shown that they can be. I'm not sure what your point is. And it is incorrect to base your analysis of future transformer performance on current transformer performance. There is a lot of ongoing research in this area and we have seen continual progress.
        soraminazuki 2y ago
        I reiterate: > This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points. "Determinism" is a word that you brought up in response to my comment, which I charitably interpreted to mean the same thing I was originally talking about. Also, it's 100% correct to analyze things based on its fundamental properties. It's absurd to criticize people for assuming 2 + 2 = 4 because "continual progress" might make it 5 in the future.
intelVISA 2y ago
sounds like it would pair well with a suitably smart compiler
- soulofmischief 2y ago
  I wrote one! It works well with cutting-edge LLMs. You feed it one or more source files that contain natural language, or stdin, and it produces a design spec, a README, and a test suite. Then it writes C code, compiles with cosmocc (for portability) and tests, in a loop, until everything is passing. All in one binary. It's been a great personal tool and I plan to open source it soon.

soulofmischief 2y ago

No, the key difference is that an engineer becomes more product-oriented, and the technicalities of the implementation are deprioritized. It is a different paradigm, in the same way that a high-level language like JavaScript handles a lot of low-level stuff for me.

soraminazuki 2y ago
A programming language implementation produces results that are controllable, reproducible, and well-defined. An LLM has none of those properties, which makes the comparison moot. Having an LLM make up underspecified details willy-nilly, or worse, ignore clear instructions is very different from programming languages "handling a lot of low-level stuff."
- soulofmischief 2y ago
  [citation needed] You can set temperature to 0 in many LLMs and get deterministic results (on the same hardware, given floating-point shenanigans). You can provide a well-defined spec and test suite. You can constrain and control the output.
  - soraminazuki 2y ago
    LLMs produce deterministic results? Now, that's a big [citation needed]. Where can I find the specs? Edit: This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points.
    - soulofmischief 2y ago
      I'd recommend learning how transformers work, and the concept of temperature. I don't think I need to cite information that is broadly and readily available, but here: https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... I also qualified the requirement of needing the same hardware, due to FP shenanigans. I could further clarify that you need the same stack (pytorch, tensorflow, etc)
      - soraminazuki 2y ago
        This gcc script that I created below is just as "deterministic" as an LLM. It produces the same result every time. Doesn't make it useful though. echo '#!/usr/bin/env bash' > gcc echo 'cat <<EOF' >> gcc openssl rand -base64 100 >> gcc echo 'EOF' >> gcc chmod +x gcc Also, how transformers work is not a spec of the LLM that anyone can use to learn how LLM produces code. It's no gcc source code.
        soulofmischief 2y ago
        You claimed they weren't deterministic, I have shown that they can be. I'm not sure what your point is. And it is incorrect to base your analysis of future transformer performance on current transformer performance. There is a lot of ongoing research in this area and we have seen continual progress.
        soraminazuki 2y ago
        I reiterate: > This is assuming by "deterministic," you mean the same thing I said about programming language implementations being "controllable, reproducible, and well-defined." If you mean it produces random but same results for the same inputs, then you haven't made any meaningful points. "Determinism" is a word that you brought up in response to my comment, which I charitably interpreted to mean the same thing I was originally talking about. Also, it's 100% correct to analyze things based on its fundamental properties. It's absurd to criticize people for assuming 2 + 2 = 4 because "continual progress" might make it 5 in the future.

intelVISA 2y ago

sounds like it would pair well with a suitably smart compiler

soulofmischief 2y ago
I wrote one! It works well with cutting-edge LLMs. You feed it one or more source files that contain natural language, or stdin, and it produces a design spec, a README, and a test suite. Then it writes C code, compiles with cosmocc (for portability) and tests, in a loop, until everything is passing. All in one binary. It's been a great personal tool and I plan to open source it soon.