3 ms·

I’m skeptical of this claim, where is he getting it from? “Elixir’s foundations in functional programming…make it easier for large language models to reason ab

I’m skeptical of this claim, where is he getting it from? “Elixir’s foundations in functional programming…make it easier for large language models to reason about, generate, and test code. LLMs struggle with imperative codebases full of side effects, hidden state, and indirection…” It seems a larger factor in LLM performance on a given language is the amount of code available to train on.

pmarreck 1y ago
I've found (specifically using Idris 2, but any functional language with some typing should work) LLM's to write better code in these restrictive languages simply because there are significantly fewer ways to screw up that are possible without causing an error upfront. In fact, I would even go so far as to say that LLM code assistance may cause a boost in interest in those languages.
- WhitneyLand 1y ago
  If you structure your prompt around small, stateless functions with explicit inputs/output, answers may look cleaner, but that advantage is not at the language level. You can do the same in TypeScript et al and get the same results. I don’t doubt that anecdotally you’ve had favorable results. However, I’m not aware of any research that backs up this claim. Or forget research - I’m not aware of any simple examples / prompts that would show it in a reproducible way across different LLMs. Corpus size tends to dominate performance.
  - pmarreck 1y ago
    > I’m not aware of any research that backs up this claim These things are literally days old. Claude 4, which is arguably the first truly useful LLM code assistant, released in May. Scientific research proceeds at a glacial pace. I'm just speaking from anecdotal experience. It may be a perceptual illusion, or it may be a real thing. Just my data point. One could deduce that restrictive languages would naturally result in fewer runtime bugs produced by LLM's since they do the exact same thing for people. For example, Elm is literally designed to produce NO runtime errors, so coding Elm in an LLM (which I haven't done) would ostensibly result in almost all errors being caught upfront. This is not to say that other types of logic errors, redundancy, etc. wouldn't slip through without human intervention.

pmarreck 1y ago

I've found (specifically using Idris 2, but any functional language with some typing should work) LLM's to write better code in these restrictive languages simply because there are significantly fewer ways to screw up that are possible without causing an error upfront. In fact, I would even go so far as to say that LLM code assistance may cause a boost in interest in those languages.

WhitneyLand 1y ago
If you structure your prompt around small, stateless functions with explicit inputs/output, answers may look cleaner, but that advantage is not at the language level. You can do the same in TypeScript et al and get the same results. I don’t doubt that anecdotally you’ve had favorable results. However, I’m not aware of any research that backs up this claim. Or forget research - I’m not aware of any simple examples / prompts that would show it in a reproducible way across different LLMs. Corpus size tends to dominate performance.
- pmarreck 1y ago
  > I’m not aware of any research that backs up this claim These things are literally days old. Claude 4, which is arguably the first truly useful LLM code assistant, released in May. Scientific research proceeds at a glacial pace. I'm just speaking from anecdotal experience. It may be a perceptual illusion, or it may be a real thing. Just my data point. One could deduce that restrictive languages would naturally result in fewer runtime bugs produced by LLM's since they do the exact same thing for people. For example, Elm is literally designed to produce NO runtime errors, so coding Elm in an LLM (which I haven't done) would ostensibly result in almost all errors being caught upfront. This is not to say that other types of logic errors, redundancy, etc. wouldn't slip through without human intervention.