6 ms·

Strong typing has clearly won. However, verbose typing is likely a negative for LLMs. Algorithms written in "pseudo-code", aka a higher level language without

Strong typing has clearly won. However, verbose typing is likely a negative for LLMs. Algorithms written in "pseudo-code", aka a higher level language without type information, are far more readable to a human, and thus likely an LLM too. In regards to control flow and general concept of what code is doing, types provide very little info over well named variables. In fact they often impair understanding by breaking up logic with implementation details. I'd be curious to see some experiments around this, but I'd guess strongly typed languages where the type information is mostly hidden/inferred would have better generation accuracy from a semantics perspective (and likely worse from a type safety perspective, but can be corrected on compile/retry)

Geniuzz 5mo ago
> Algorithms written in "pseudo-code", aka a higher level language without type information, are far more readable to a human, and thus likely an LLM too. What’s the basis of this claim? There are many many more lines of code LLM’s are trained versus pseudo-code. Also I agree, anecdotally the self-correction is key benefit from static types. If there is a mistake, it is caught at compile time and not at runtime.
- adam_arthur 5mo ago
  It seems clear to me from first principles. Humans are trained on human language. LLMs are trained on human language. Thus something that is easier for a human to understand is likely easier for an LLM to understand. That higher level language with well named variables reads more comprehensibly than code:VERB with:PREPOSITION types:NOUN, intermixed:ADJECTIVE, stems:VERB from:PREPOSITION first:ADJECTIVE principles:NOUN too:ADVERB
  - ModernMech 5mo ago
    As far as the AI is concerned, it's more like Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo versus Buffalo:PN buffalo:N Buffalo:PN buffalo:N buffalo:V buffalo:V Buffalo:PN buffalo:N I think the second one makes much more sense.
    - adam_arthur 5mo ago
      In the rare case that all your concepts use the exact same descriptive word, you are probably right! The majority of the time you can infer the type from reading well written code (to the extent that the shape of the type matters in the context of that piece of code)
      - deleted 5mo ago
        [deleted]
      - deleted 5mo ago
        [deleted]
      - ModernMech 5mo ago
        If the type can be inferred by the reader it should be inferred by the type system and at least be available to the LLM as a query. But we're also talking about dynamic languages in which type cannot be inferred until runtime. What's the type of x? x = y + z Well that depends on the types of y and z, which themselves may depend on the types of other operands, which themselves may not be known until the program actually runs. All that inference takes a lot of thinking, which takes tokens, which cost money. Why not just write the types down? Although we call these things "inference engines" they're really pattern matching explicit tokens, so it's better to actually write down the types so they can be pattern matched than to figure them out at inference time.
      - cocophone 5mo ago
        You are basically rehashing the false beliefs of the codeless programming camp. Human language that is 99% correct is a standing ovation for a speech writer while it is paying a cyber ransom as the software maker.
  - ethanlipson 5mo ago
    For models as complex as these I'm not confident we can apply arguments from first principles; we could just as easily argue that type information is helpful, from first principles. What is much more useful is empirical evidence, and AutoCodeBench [1] found that LLMs are most proficient in Elixir (dynamic) followed by Kotlin (static), with Rust and PHP at the bottom. So it would seem like, as of publication, typing style doesn't really matter! [1] https://autocodebench.github.io/ https://autocodebench.github.io/
vmg12 5mo ago
> types provide very little info over well named variables Types guarantee invariants at compile time, adding type info to a variable name is just a prayer that the next human or robot will enforce the invariants with respect to that type when it matters. This is like saying you don't need a saw stop because you should just avoid sticking your hand in the saw blade.
cocophone 5mo ago
I think Java shows that too much naming is a horrible idea even if there is a good type layer. It's exceptionally easy to have homonyms and other problems that feed errors in LLMs and if every mix of nonsense runs that's all the worse. Add to that attempts to put many English words together and there are no way points left. Everything is an exhausting essay by Dickens where two very long sentences are subtly different (to look at but not in results.)
mvdtnz 5mo ago
This is a lot of speculation with absolutely nothing backing it.
MaxBarraclough 5mo ago
Strong typing is not a synonym for static typing, it refers to a different aspect of type safety. Static typing is, roughly, where variables and expressions have fixed types that can be determined ahead of execution. Strong typing means the language doesn't offer implicit type conversions. Python is dynamically typed, i.e. not statically typed, and strongly typed. (Ignoring its type annotations feature, of course.)

Geniuzz 5mo ago

> Algorithms written in "pseudo-code", aka a higher level language without type information, are far more readable to a human, and thus likely an LLM too. What’s the basis of this claim? There are many many more lines of code LLM’s are trained versus pseudo-code. Also I agree, anecdotally the self-correction is key benefit from static types. If there is a mistake, it is caught at compile time and not at runtime.

adam_arthur 5mo ago
It seems clear to me from first principles. Humans are trained on human language. LLMs are trained on human language. Thus something that is easier for a human to understand is likely easier for an LLM to understand. That higher level language with well named variables reads more comprehensibly than code:VERB with:PREPOSITION types:NOUN, intermixed:ADJECTIVE, stems:VERB from:PREPOSITION first:ADJECTIVE principles:NOUN too:ADVERB
- ModernMech 5mo ago
  As far as the AI is concerned, it's more like Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo versus Buffalo:PN buffalo:N Buffalo:PN buffalo:N buffalo:V buffalo:V Buffalo:PN buffalo:N I think the second one makes much more sense.
  - adam_arthur 5mo ago
    In the rare case that all your concepts use the exact same descriptive word, you are probably right! The majority of the time you can infer the type from reading well written code (to the extent that the shape of the type matters in the context of that piece of code)
    - deleted 5mo ago
      [deleted]
    - deleted 5mo ago
      [deleted]
    - ModernMech 5mo ago
      If the type can be inferred by the reader it should be inferred by the type system and at least be available to the LLM as a query. But we're also talking about dynamic languages in which type cannot be inferred until runtime. What's the type of x? x = y + z Well that depends on the types of y and z, which themselves may depend on the types of other operands, which themselves may not be known until the program actually runs. All that inference takes a lot of thinking, which takes tokens, which cost money. Why not just write the types down? Although we call these things "inference engines" they're really pattern matching explicit tokens, so it's better to actually write down the types so they can be pattern matched than to figure them out at inference time.
    - cocophone 5mo ago
      You are basically rehashing the false beliefs of the codeless programming camp. Human language that is 99% correct is a standing ovation for a speech writer while it is paying a cyber ransom as the software maker.
- ethanlipson 5mo ago
  For models as complex as these I'm not confident we can apply arguments from first principles; we could just as easily argue that type information is helpful, from first principles. What is much more useful is empirical evidence, and AutoCodeBench [1] found that LLMs are most proficient in Elixir (dynamic) followed by Kotlin (static), with Rust and PHP at the bottom. So it would seem like, as of publication, typing style doesn't really matter! [1] https://autocodebench.github.io/ https://autocodebench.github.io/

vmg12 5mo ago

> types provide very little info over well named variables Types guarantee invariants at compile time, adding type info to a variable name is just a prayer that the next human or robot will enforce the invariants with respect to that type when it matters. This is like saying you don't need a saw stop because you should just avoid sticking your hand in the saw blade.

cocophone 5mo ago

I think Java shows that too much naming is a horrible idea even if there is a good type layer. It's exceptionally easy to have homonyms and other problems that feed errors in LLMs and if every mix of nonsense runs that's all the worse. Add to that attempts to put many English words together and there are no way points left. Everything is an exhausting essay by Dickens where two very long sentences are subtly different (to look at but not in results.)

mvdtnz 5mo ago

This is a lot of speculation with absolutely nothing backing it.

MaxBarraclough 5mo ago

Strong typing is not a synonym for static typing, it refers to a different aspect of type safety. Static typing is, roughly, where variables and expressions have fixed types that can be determined ahead of execution. Strong typing means the language doesn't offer implicit type conversions. Python is dynamically typed, i.e. not statically typed, and strongly typed. (Ignoring its type annotations feature, of course.)