19 ms·

Which programming languages are most token-efficient?

115 points by tehnub 9mo ago

btbytes 9mo ago
Not surprisingly, it is J [1], an APL dialect. [1] https://www.jsoftware.com/ https://www.jsoftware.com/
- eimrine 9mo ago
  I knew it without the reading. But having each system call in 2 versions not even closely related to each other (monadic/diadic) requires me to have a hard time doing learning. I very appreciate this language for shortness but this kind of shortness might annoy.
epolanski 9mo ago
I doubt this to be a meaningful metric for anything but code exploration in a larger codebase. E.g. when it comes to authoring code, C, which comes language, is by far one of the languages that LLMs excel most at.
- anishgupta 9mo ago
  I guess it also depends on which dataset LLM was trained on. Rare or niche languages get fragmented into more tokens even if the code itself is short. So two languages with the same number of characters can produce very different token counts because one aligns with what the model has seen millions of times and the other does not.
HarHarVeryFunny 9mo ago
I don't think context size is really the limit for larger codebases - it's more about how you use that context. Claude Code makes some efforts to reduce context size, but at the end of the day is loading entire source files into context (then keeping them there until told to remove them, or context is compacted). One of the major wins is to run subagents for some tasks, that use their own context rather than loading more into CCs own context. Cursor makes more efficient use of context by building a vector database of code chunks, then only loading matching chunks into context (I believe it does this for Composer/agentic use as well as for tab/autocomplete). One of the more obvious ways to reduce context use in a larger multi-module codebase would be to take advantage of the split between small module definition (e.g. C++ .h files) and large module implementations (.cpp files). Generally you'd only need to load module interfaces/definitions into context if you are working on code that uses the module, and Cursor's chunked approach can reduce that further. For whole codebase overview a language server can help locate things, and one could use the AI to itself generate shortish summaries/overviews of source files and the codebase and structure, similar to what a human developer might keep in their head, rather than repeatedly reading entire source files for code that isn't actually being modified. It seems we're really in the early days of agentic coding tools, and they have a lot of room to get better and more efficient.
- CuriouslyC 9mo ago
  The approaches used by Claude Code and Cursor are inefficient. It's possible to calculate a covering set for a piece of code and provide that to an agent directly via a tool, and it turns out that this can reduce context usage in SWE-bench style tasks by >90% over RAG and grep/read. If you're interested in learning more, https://github.com/sibyllinesoft/scribe https://github.com/sibyllinesoft/scribe
  - trueno 9mo ago
    Like most LLM-made readme's and the six bajillion AI/agentic/llm tools now on Github I can barely get a grasp on what I'm looking at here, or how to use it practically. > Smart code bundler that turns repositories into optimized code bundles meeting a token budget in milliseconds Ok. So it's a tool, do I use it on my repo once? Then what? Do I use it as I go, does it sit somewhere accessible to something like Claude Code and the onus is on me to direct Claude to use this to search files instead of his out of box workflow ? I can see some CLI examples, what should I do with that where does that fit into what people are using with cursor / claude / gemini etc ? This is the part I've been trying to hammer home about LLM created stuff. It leaves us with vague not well-understood outcomes that might do something. People are shipping/delivering things they don't even understand now and they often times can't speak to what their thing does with an acceptable level of authority. I'm not against creating tools with LLM's but I'm actually pretty against people creating the basic readme with LLM's. Wanna make a tool in an LLM? More power to you. But make sure you understand what was made, because we need humans in here telling other humans how to use it, because LLMs flat out lose the plot over the course of a large project and I think a big issue is LLM's can sometimes be more eloquent at writing than a lot of people can, so they opt for the LLM-generated readme. But as someone who would maybe consider using something like this, I see that readme and it just looks like every claude code thing I've put together to date which is to say I've done some seemingly impossible things with Claude only to find that his ability to recap the entirety of it just ended up in a whole lot of seemingly meaningful words and phrases and sentences that actually paint a super disjointed picture of what exactly a repo is about.
    - 9mo ago
lngnmn2 9mo ago
[dead]
bicx 9mo ago
Realistically, it’s also a function of how many iterations it takes for an AI agent to correctly solve a problem with a given language. I’d imagine most AI agents would frequently have to redo J or F# code, as they are fairly uncommon languages with much smaller training set than JavaScript or Python.
- Jacques2Marais 9mo ago
  I can say that for F# this has been mostly true up until quite recently. We use F# at work and were mostly unable to use agents like Claude Code up until the release of Opus 4.5, which seems to know F# quite well.
  - Smaug123 9mo ago
    (I concur, Opus 4.5 is quite capable of writing F#.)
yeasku 9mo ago
[dead]
switchbak 9mo ago
I would expect that we’ll end up compressing (or whatever term you would use) this at some point so many of those syntactical differences will not be as significant. But I would love for more expressive and compact languages to do better, selfish as I am. But I think training data size is more of a factor, and we won’t be all moving up Clojure any time soon.
- eigenspace 9mo ago
  I can't speak to Clojure, but I will say that LLMs are actually surprisingly good at writing and understanding Julia code compared to some languages that have a much larger training corpus to pull from.
bri-holt 9mo ago
I suspect DB queries will also benefit from token-efficient query languages as RAG queries grow exponentially. I've been working on one such language that is emitted in a token-efficient IR and compiles to SQL. https://memelang.net/ https://memelang.net/
gpm 9mo ago
It strikes me that more tokens likely give the LLM more time/space to "think". Also that more redundant tokens, like local type declarations instead of type inference from far away, likely often reduce the portion of the code LLMs (and humans) have to read. So I'm not convinced this is either the right metric, or even if you got the right metric that it's a metric you want to minimize.
- limoce 9mo ago
  I think separating thinking tokens from "representing" tokens might be a better approach, like what those thinking models does
- make3 9mo ago
  With Chain of Thoughts (text thinking), the models can already use as much compute as they want in any language (determined by reinforcement learning training)
  - gpm 9mo ago
    I'm not convinced that thinking tokens - which sort of have to serve a specific chain of thought purpose - are interchangeable with input tokens during which give the model compute without having it add new text. For a very imperfect human analogy, it feels like saying "a student can spend as much time thinking about the text as they want, so the textbook can be extremely terse". Definitely just gut feelings though - not well tested or anything. I could be wrong.
    - make3 9mo ago
      We could definitely use RL to add blank, invisible "<thinking>" tokens whenever the model thinks it should. Or just allow it to say "Hmm." lol. Easy to test from a technical perspective is all I'm saying, and not a bad idea.
didip 9mo ago
Does it account for errors generated from Runtime bugs which caused rerunning of prompts? Because that’s what happened in the real world when generating a bunch of untyped Python code.
efitz 9mo ago
This is interesting research; thank you for doing it. I am not sure token efficiency is an interesting problem in the long term, though. And in the short term I wonder if prompts could be pre-compiled to “compressed tokens”; the idea would be to use a smaller number of tokens to represent a frequently needed concept; kind of like LZ compression. Or maybe token compression becomes a feature of future models optimized for specific tasks. I was wondering last year if it would be worthwhile trying to create a language that was especially LLM-friendly, eg that embedded more context in the language structure. The idea is to make more of the program and the thinking behind it, explicit to the LLM but in a programming language style to eliminate the ambiguity of natural language (one could just use comments). Then it occurred to me that with current LLM training methodology that there’s a chicken-and-egg problem; it doesn’t start to show rewards until there is a critical mass of good code in the language for LLMs to train on.
protocolture 9mo ago
I have always had concerns about physical robots making my work less safe in the real world. But had never considered that a programming language might be created thats less human readable/auditable to enable LLMs. Scares me a bit.
- make3 9mo ago
  LLMs in their current form rely heavily on the vast amount of human data that's available, to learn from it as a first step (the second step is RL). We're not building a language for LLMs just yet.
  - energy123 9mo ago
    It's worth asking why we haven't had the AlphaZero moment for general learning yet, where no human data is needed.
    - jaggederest 9mo ago
      I think the issue is that for games and other closed-ended systems the criteria are very easy, so self-referential training is effective.
    - make3 9mo ago
      That's easy, AlphaZero had a perfect simulator of the world it existed in (chess, super easy), so it was insanely easy to run simulations of that world ad infinitum, and learn from it. It's simply not the case for the real world, you can't simulate the world perfectly and see what happens when you do things.
  - jaggederest 9mo ago
    > We're not building a language for LLMs just yet. Working on it, actually! I think it's a really interesting problem space - being efficient on tokens, readable by humans for review, strongly typed and static for reasoning purposes, and having extremely regular syntax. One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it. I expect a language like the one I'm playing with will mature enough over the next couple years that models with a knowledge cutoff around 1/2027 will probably know how to program it well enough for it to start being more viable. One of the things I plan to do is build evals so that I can validate the performance of various models on my as yet only partially baked language. I'm also using only LLMs to build out the entire infrastructure, mostly to see if it's possible.
janalsncm 9mo ago
This is kind of just a measurement of how representative a language is in the distribution of the tokenizer training. You could have a single token equal to “public static void main”.
- muyuu 9mo ago
  You could, but you wouldn't when those keywords can all change in equivalent contexts.
  - eru 9mo ago
    What do you mean? `public` might have a token by itself, even though you can have `pub` occurring in other contexts, too.
    - muyuu 9mo ago
      I meant that it wouldn't be efficient to agglomerate tokens in that way and that's why the system won't do it
  - janalsncm 9mo ago
    The BPE or wordpiece tokenization algorithm will greedily take the longest valid token prefix. So if your text starts with “public static void main” it will try to find the longest token which matches that prefix. Even if “public” is a token, it will prefer to tokenize “public static” together.
    - muyuu 9mo ago
      yes, but then you have both alternatives as tokens, which nullifies GP's argument
- moelf 9mo ago
  the most efficient languages are pretty unpopular, so this argument makes them even more efficient in reality?...
- 9mo ago
solomonb 9mo ago
I'm biased by my preferred style of programming languages but I think that pure statically typed functional languages are incredibly well suited for LLMs. The purity gives you referential transparency and static analysis powers that the LLM can leverage to stay correctly on task. The high level declarative nature and type driven development style of languages like Haskell also make it really easy for an experienced developer to review and validate the output of the LLM. Early on in the GPT era I had really bad experiences generating Haskell code with LLMs but I think that the combination of improved models, increased context size, and agentic tooling has allowed LLMs to really take advantage of functional programming.
- eru 9mo ago
  I'm inclined to agree with you in principle, but there's much, much less Haskell examples in their training corpus than for JavaScript or Python.
  - kstrauser 9mo ago
    And yet the models I've used have been great with Rust, which pales in lines of code to JavaScript (or Python or PHP or Perl or C or C++).
    - eru 9mo ago
      I've also had decent experiences with Rust recently. I haven't done enough Haskell programming in the AI era to really say. But it could be that different programming languages are a bit like different human languages for these models: when they have more than some threshold of training data, they can express their general problem solving skills in any of them? And then it's down to how much the compiler and linters can yell at them. For Rust, I regularly tell them to make `clippy::pedantic` happy (and tell me explicitly when they think that the best way to do that is via an explicit ignore annotation in the code to disable a certain warning for a specific line). Pedantic clippy is usually too.. pedantic for humans, but it seems to work reasonably well with the agents. You can also add clippy::cargo which ain't included in clippy::pedantic.
TZubiri 9mo ago
05a1be
- xigoi 9mo ago
  Given that APL was punished for using a non-ASCII character set, this would presumably also affect 05AB1E.
awesome_dude 9mo ago
I'm finding that I have to share more and more code to ensure that various standards are being kept. For example I shared some Model code with Claude and Gemini (both via web interfaces) and they both tried to put Controller code into the Model, despite me multiple times telling them that the code wasn't wanted nor needed in there. I had to (eventually) share the entire project with the models (despite them having been working with the code all along) before they would comply with my request (whilst also congratulating me on my far superior architecture..) That costs more tokens for each problem than just saying "her look at this section and work toward this goal"
- solumunus 9mo ago
  You’re severely hampering yourself by not using the CLI tools.
verdverm 9mo ago
Token efficiency is only one metric. Simplicity of syntax and semantics are another valuable one. re: tokens and session length, there are other ways to manage this than language choice. Summarization is one, something I do is to not out read_file content in the messages, but rather in the system prompt. This means that when it tries to reread after edit, we don't have two copies of the file in context. Going to 10M token sessions, keeping per turn context under 100k, working on Golang... language choice for the sake of tokens does not seem a good thing to decide based on
kozika 9mo ago
Someone has made a programming language called Sui, which is said to be designed for LLMs. However, using index-based variable names in order to "avoid typo bugs" makes it more difficult than general-purpose languages, and it also has poor token efficiency :( https://github.com/TakatoHonda/sui-lang https://github.com/TakatoHonda/sui-lang
tzahifadida 9mo ago
An agent can make summaries via Markdown files while processing. Then use that to break the problem to several issues and then tackle them one by one, even automatically, but more usually interactively. The problem is the technique now, not the llm. Yes, it costs a lot (lot) more. But, it can do it, and people work cost way more than tokens.
112233 9mo ago
If language supports comments and LLM is allowed to write them (or docstrings, or any such), there go your tokens. Plus, they will strongly "pull" the context when LLM parses it back, to the point of overriding your instructions (true story)
andersmurphy 9mo ago
Forth
nineteen999 9mo ago
Ive been doing plenty of Z80/x86_64 assembly with it, as well as a little 6502. Those are pretty terse.
Surac 9mo ago
intresting project, do you mind to explain what brought you to do that research? im a litte surprised that the more simple languages tend to use more tokens, but after thing i realizend that languages with more expressiv syntax allow to write with less "words". But i also think it is a little bit like a race of watches. who realy wants to know what watch runs faster?
johnisgood 9mo ago
Concatenative languages like Factor and Forth are very token-efficient in theory. Theoretically optimal for raw lexical density. No parentheses, no commas, no argument delimiters, just whitespace-separated words, but stack shuffling can add overhead for complex data flow, unless you use "locals" in Factor, for example. C is surprisingly efficient as well. Minimal keywords, terse syntax, single-character operators. Not much boilerplate, and the core logic is dense. I think the worst languages are Java, C#, and Rust (lifetime annotations, verbose generics). In my opinion, C or Go for imperative code, Factor / Forth if the model knows them well.
- Bootvis 9mo ago
  I understand your logic but I found LLM's to be quite strong at C#. It makes little mistakes and the mistakes seem related to the complexity of what I'm doing, not the language itself.
  - johnisgood 9mo ago
    See https://news.ycombinator.com/item?id=46586312 https://news.ycombinator.com/item?id=46586312. I cannot speak much for C#, but you may be right. Claude's Opus is really good.
- Smaug123 9mo ago
  Is that statement about C based on anything in particular? C was 18th of all the languages in the article's chart (the worst!), which I'd guess was due to the absence of a standard library.
  - johnisgood 9mo ago
    Fair point. There is a distinction between syntactic efficiency (C is terse) and task-completion efficiency (what the benchmark likely measured). If the tasks involved string manipulation, hash maps, JSON, etc. then C pays a massive token tax because you are implementing what other languages provide in stdlib. Python has dict and json.loads(), C has malloc and strcmp. So: C tokenizes efficiently for equivalent logic, but stdlib poverty makes it expensive for typical benchmark tasks. Same applies to Factor/Forth, arguably worse.
aleph_minus_one 9mo ago
Relevant: On https://danuker.go.ro/programming-languages.html https://danuker.go.ro/programming-languages.html you can find charts of popularity (TIOBE) vs code density for various programming languages together with which programming languages are Pareto-optimal regarding these two criteria.
HtmlProgrammer 9mo ago
> I then told Claude Code to suggest a selection of the most popular programming languages If you’re going to write an article atleast do the base research yourself man
jwr 9mo ago
I program mostly in Clojure and I expected it to be near the top, as it tends to be very concise and expressive (qualities I really admire). I am getting excellent results from Claude Code (Opus 4.5), and I think this might be one of the reasons. I'm using Claude with a large code base and the token-efficiency of Clojure might help with fitting more into the context window.
- kaliszad 9mo ago
  I also program a lot in Clojure/Script. Do you also consider thinking token and the number of iterations in the token efficiency?
  - jwr 9mo ago
    I don't think thinking tokens are affected, as LLMs "think" mostly in plain language, with occasional code snippets.
    - kaliszad 9mo ago
      I would assume for certain problems LLMs have a solution readily available for JavaScript/ TypeScript or similarly popular languages but not for Clojure/Script. Therefore my thinking was that the process of getting to a workable solution would be longer and more expensive in terms of tokens. I however don't have any relevant data on this so I may just be wrong.
nige123 9mo ago
Raku
torginus 9mo ago
This confirms my personal experience with switching to Go from C# - despite the e framework and language being MUCH simpler, the code usually ends up the same length. C# often has a 'nice' and 'performant' way of doing things (for example, strings are nice, but they allocate and are UTF16, but ReadOnlySpan<byte> is faster for UTF8, and can reuse buffers), the performant syntax often ends up being very verbose, with the nice syntax being barely shorter than Go's. Go also does the right thing by default, and its strings are basically array slices into UTF8 byte arrays.
- neonsunset 9mo ago
  [dead]
thw_9a83c 9mo ago
There is one class of languages missing in the comparison: Programming golf languages: E.g. Japt [1], Pyth [2] or Jelly [3]. Update: I noticed that the author mentions that "APL's famous terseness isn't a plus for LLMs." Isn't that just a design limitation of the LLM tokenizers? [1]: https://github.com/ETHproductions/japt https://github.com/ETHproductions/japt [2]: https://github.com/isaacg1/pyth https://github.com/isaacg1/pyth [3]: https://github.com/DennisMitchell/jellylanguage https://github.com/DennisMitchell/jellylanguage
tmtvl 9mo ago
I kinda (but not really because I don't much care about tokens and don't really know anything about models) wonder about Common Lisp. There's probably far fewer examples of CL code in any training sets than Clojure or Python or whatever, but it could still be somewhat interesting.
singularity2001 9mo ago
That's why I love Julia so much. Also it's semi-static?
- eigenspace 9mo ago
  Semantically, julia is a fully dynamic language. But the trick is that it does this by recognizing that being static is a constraint on a dynamic language, so it implements dynamic typing by stitching together islands of statically typed code.
6510 9mo ago
Curious about FORTH now
daft_pink 9mo ago
Wow, I’m really shocked that something like Javascript and Typescript which models tend to prefer wasn’t the most efficient.
zhisme 9mo ago
ruby. that's quite interesting, maybe because it reads mostly like plain english?
reena_signalhq 9mo ago
[dead]

115 points by tehnub 9mo ago

btbytes 9mo ago

Not surprisingly, it is J [1], an APL dialect. [1] https://www.jsoftware.com/ https://www.jsoftware.com/

eimrine 9mo ago
I knew it without the reading. But having each system call in 2 versions not even closely related to each other (monadic/diadic) requires me to have a hard time doing learning. I very appreciate this language for shortness but this kind of shortness might annoy.

epolanski 9mo ago

I doubt this to be a meaningful metric for anything but code exploration in a larger codebase. E.g. when it comes to authoring code, C, which comes language, is by far one of the languages that LLMs excel most at.

anishgupta 9mo ago
I guess it also depends on which dataset LLM was trained on. Rare or niche languages get fragmented into more tokens even if the code itself is short. So two languages with the same number of characters can produce very different token counts because one aligns with what the model has seen millions of times and the other does not.

HarHarVeryFunny 9mo ago

I don't think context size is really the limit for larger codebases - it's more about how you use that context. Claude Code makes some efforts to reduce context size, but at the end of the day is loading entire source files into context (then keeping them there until told to remove them, or context is compacted). One of the major wins is to run subagents for some tasks, that use their own context rather than loading more into CCs own context. Cursor makes more efficient use of context by building a vector database of code chunks, then only loading matching chunks into context (I believe it does this for Composer/agentic use as well as for tab/autocomplete). One of the more obvious ways to reduce context use in a larger multi-module codebase would be to take advantage of the split between small module definition (e.g. C++ .h files) and large module implementations (.cpp files). Generally you'd only need to load module interfaces/definitions into context if you are working on code that uses the module, and Cursor's chunked approach can reduce that further. For whole codebase overview a language server can help locate things, and one could use the AI to itself generate shortish summaries/overviews of source files and the codebase and structure, similar to what a human developer might keep in their head, rather than repeatedly reading entire source files for code that isn't actually being modified. It seems we're really in the early days of agentic coding tools, and they have a lot of room to get better and more efficient.

CuriouslyC 9mo ago
The approaches used by Claude Code and Cursor are inefficient. It's possible to calculate a covering set for a piece of code and provide that to an agent directly via a tool, and it turns out that this can reduce context usage in SWE-bench style tasks by >90% over RAG and grep/read. If you're interested in learning more, https://github.com/sibyllinesoft/scribe https://github.com/sibyllinesoft/scribe
- trueno 9mo ago
  Like most LLM-made readme's and the six bajillion AI/agentic/llm tools now on Github I can barely get a grasp on what I'm looking at here, or how to use it practically. > Smart code bundler that turns repositories into optimized code bundles meeting a token budget in milliseconds Ok. So it's a tool, do I use it on my repo once? Then what? Do I use it as I go, does it sit somewhere accessible to something like Claude Code and the onus is on me to direct Claude to use this to search files instead of his out of box workflow ? I can see some CLI examples, what should I do with that where does that fit into what people are using with cursor / claude / gemini etc ? This is the part I've been trying to hammer home about LLM created stuff. It leaves us with vague not well-understood outcomes that might do something. People are shipping/delivering things they don't even understand now and they often times can't speak to what their thing does with an acceptable level of authority. I'm not against creating tools with LLM's but I'm actually pretty against people creating the basic readme with LLM's. Wanna make a tool in an LLM? More power to you. But make sure you understand what was made, because we need humans in here telling other humans how to use it, because LLMs flat out lose the plot over the course of a large project and I think a big issue is LLM's can sometimes be more eloquent at writing than a lot of people can, so they opt for the LLM-generated readme. But as someone who would maybe consider using something like this, I see that readme and it just looks like every claude code thing I've put together to date which is to say I've done some seemingly impossible things with Claude only to find that his ability to recap the entirety of it just ended up in a whole lot of seemingly meaningful words and phrases and sentences that actually paint a super disjointed picture of what exactly a repo is about.
  - 9mo ago

lngnmn2 9mo ago

[dead]

bicx 9mo ago

Realistically, it’s also a function of how many iterations it takes for an AI agent to correctly solve a problem with a given language. I’d imagine most AI agents would frequently have to redo J or F# code, as they are fairly uncommon languages with much smaller training set than JavaScript or Python.

Jacques2Marais 9mo ago
I can say that for F# this has been mostly true up until quite recently. We use F# at work and were mostly unable to use agents like Claude Code up until the release of Opus 4.5, which seems to know F# quite well.
- Smaug123 9mo ago
  (I concur, Opus 4.5 is quite capable of writing F#.)

yeasku 9mo ago

[dead]

switchbak 9mo ago

I would expect that we’ll end up compressing (or whatever term you would use) this at some point so many of those syntactical differences will not be as significant. But I would love for more expressive and compact languages to do better, selfish as I am. But I think training data size is more of a factor, and we won’t be all moving up Clojure any time soon.

eigenspace 9mo ago
I can't speak to Clojure, but I will say that LLMs are actually surprisingly good at writing and understanding Julia code compared to some languages that have a much larger training corpus to pull from.

bri-holt 9mo ago

I suspect DB queries will also benefit from token-efficient query languages as RAG queries grow exponentially. I've been working on one such language that is emitted in a token-efficient IR and compiles to SQL. https://memelang.net/ https://memelang.net/

gpm 9mo ago

It strikes me that more tokens likely give the LLM more time/space to "think". Also that more redundant tokens, like local type declarations instead of type inference from far away, likely often reduce the portion of the code LLMs (and humans) have to read. So I'm not convinced this is either the right metric, or even if you got the right metric that it's a metric you want to minimize.

limoce 9mo ago
I think separating thinking tokens from "representing" tokens might be a better approach, like what those thinking models does
make3 9mo ago
With Chain of Thoughts (text thinking), the models can already use as much compute as they want in any language (determined by reinforcement learning training)
- gpm 9mo ago
  I'm not convinced that thinking tokens - which sort of have to serve a specific chain of thought purpose - are interchangeable with input tokens during which give the model compute without having it add new text. For a very imperfect human analogy, it feels like saying "a student can spend as much time thinking about the text as they want, so the textbook can be extremely terse". Definitely just gut feelings though - not well tested or anything. I could be wrong.
  - make3 9mo ago
    We could definitely use RL to add blank, invisible "<thinking>" tokens whenever the model thinks it should. Or just allow it to say "Hmm." lol. Easy to test from a technical perspective is all I'm saying, and not a bad idea.

didip 9mo ago

Does it account for errors generated from Runtime bugs which caused rerunning of prompts? Because that’s what happened in the real world when generating a bunch of untyped Python code.

efitz 9mo ago

This is interesting research; thank you for doing it. I am not sure token efficiency is an interesting problem in the long term, though. And in the short term I wonder if prompts could be pre-compiled to “compressed tokens”; the idea would be to use a smaller number of tokens to represent a frequently needed concept; kind of like LZ compression. Or maybe token compression becomes a feature of future models optimized for specific tasks. I was wondering last year if it would be worthwhile trying to create a language that was especially LLM-friendly, eg that embedded more context in the language structure. The idea is to make more of the program and the thinking behind it, explicit to the LLM but in a programming language style to eliminate the ambiguity of natural language (one could just use comments). Then it occurred to me that with current LLM training methodology that there’s a chicken-and-egg problem; it doesn’t start to show rewards until there is a critical mass of good code in the language for LLMs to train on.

protocolture 9mo ago

I have always had concerns about physical robots making my work less safe in the real world. But had never considered that a programming language might be created thats less human readable/auditable to enable LLMs. Scares me a bit.

make3 9mo ago
LLMs in their current form rely heavily on the vast amount of human data that's available, to learn from it as a first step (the second step is RL). We're not building a language for LLMs just yet.
- energy123 9mo ago
  It's worth asking why we haven't had the AlphaZero moment for general learning yet, where no human data is needed.
  - jaggederest 9mo ago
    I think the issue is that for games and other closed-ended systems the criteria are very easy, so self-referential training is effective.
  - make3 9mo ago
    That's easy, AlphaZero had a perfect simulator of the world it existed in (chess, super easy), so it was insanely easy to run simulations of that world ad infinitum, and learn from it. It's simply not the case for the real world, you can't simulate the world perfectly and see what happens when you do things.
- jaggederest 9mo ago
  > We're not building a language for LLMs just yet. Working on it, actually! I think it's a really interesting problem space - being efficient on tokens, readable by humans for review, strongly typed and static for reasoning purposes, and having extremely regular syntax. One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it. I expect a language like the one I'm playing with will mature enough over the next couple years that models with a knowledge cutoff around 1/2027 will probably know how to program it well enough for it to start being more viable. One of the things I plan to do is build evals so that I can validate the performance of various models on my as yet only partially baked language. I'm also using only LLMs to build out the entire infrastructure, mostly to see if it's possible.

janalsncm 9mo ago

This is kind of just a measurement of how representative a language is in the distribution of the tokenizer training. You could have a single token equal to “public static void main”.

muyuu 9mo ago
You could, but you wouldn't when those keywords can all change in equivalent contexts.
- eru 9mo ago
  What do you mean? `public` might have a token by itself, even though you can have `pub` occurring in other contexts, too.
  - muyuu 9mo ago
    I meant that it wouldn't be efficient to agglomerate tokens in that way and that's why the system won't do it
- janalsncm 9mo ago
  The BPE or wordpiece tokenization algorithm will greedily take the longest valid token prefix. So if your text starts with “public static void main” it will try to find the longest token which matches that prefix. Even if “public” is a token, it will prefer to tokenize “public static” together.
  - muyuu 9mo ago
    yes, but then you have both alternatives as tokens, which nullifies GP's argument
moelf 9mo ago
the most efficient languages are pretty unpopular, so this argument makes them even more efficient in reality?...
9mo ago

solomonb 9mo ago

I'm biased by my preferred style of programming languages but I think that pure statically typed functional languages are incredibly well suited for LLMs. The purity gives you referential transparency and static analysis powers that the LLM can leverage to stay correctly on task. The high level declarative nature and type driven development style of languages like Haskell also make it really easy for an experienced developer to review and validate the output of the LLM. Early on in the GPT era I had really bad experiences generating Haskell code with LLMs but I think that the combination of improved models, increased context size, and agentic tooling has allowed LLMs to really take advantage of functional programming.

eru 9mo ago
I'm inclined to agree with you in principle, but there's much, much less Haskell examples in their training corpus than for JavaScript or Python.
- kstrauser 9mo ago
  And yet the models I've used have been great with Rust, which pales in lines of code to JavaScript (or Python or PHP or Perl or C or C++).
  - eru 9mo ago
    I've also had decent experiences with Rust recently. I haven't done enough Haskell programming in the AI era to really say. But it could be that different programming languages are a bit like different human languages for these models: when they have more than some threshold of training data, they can express their general problem solving skills in any of them? And then it's down to how much the compiler and linters can yell at them. For Rust, I regularly tell them to make `clippy::pedantic` happy (and tell me explicitly when they think that the best way to do that is via an explicit ignore annotation in the code to disable a certain warning for a specific line). Pedantic clippy is usually too.. pedantic for humans, but it seems to work reasonably well with the agents. You can also add clippy::cargo which ain't included in clippy::pedantic.

TZubiri 9mo ago

05a1be

xigoi 9mo ago
Given that APL was punished for using a non-ASCII character set, this would presumably also affect 05AB1E.

awesome_dude 9mo ago

I'm finding that I have to share more and more code to ensure that various standards are being kept. For example I shared some Model code with Claude and Gemini (both via web interfaces) and they both tried to put Controller code into the Model, despite me multiple times telling them that the code wasn't wanted nor needed in there. I had to (eventually) share the entire project with the models (despite them having been working with the code all along) before they would comply with my request (whilst also congratulating me on my far superior architecture..) That costs more tokens for each problem than just saying "her look at this section and work toward this goal"

solumunus 9mo ago
You’re severely hampering yourself by not using the CLI tools.

verdverm 9mo ago

Token efficiency is only one metric. Simplicity of syntax and semantics are another valuable one. re: tokens and session length, there are other ways to manage this than language choice. Summarization is one, something I do is to not out read_file content in the messages, but rather in the system prompt. This means that when it tries to reread after edit, we don't have two copies of the file in context. Going to 10M token sessions, keeping per turn context under 100k, working on Golang... language choice for the sake of tokens does not seem a good thing to decide based on

kozika 9mo ago

Someone has made a programming language called Sui, which is said to be designed for LLMs. However, using index-based variable names in order to "avoid typo bugs" makes it more difficult than general-purpose languages, and it also has poor token efficiency :( https://github.com/TakatoHonda/sui-lang https://github.com/TakatoHonda/sui-lang

tzahifadida 9mo ago

An agent can make summaries via Markdown files while processing. Then use that to break the problem to several issues and then tackle them one by one, even automatically, but more usually interactively. The problem is the technique now, not the llm. Yes, it costs a lot (lot) more. But, it can do it, and people work cost way more than tokens.

112233 9mo ago

If language supports comments and LLM is allowed to write them (or docstrings, or any such), there go your tokens. Plus, they will strongly "pull" the context when LLM parses it back, to the point of overriding your instructions (true story)

andersmurphy 9mo ago

Forth

nineteen999 9mo ago

Ive been doing plenty of Z80/x86_64 assembly with it, as well as a little 6502. Those are pretty terse.

Surac 9mo ago

intresting project, do you mind to explain what brought you to do that research? im a litte surprised that the more simple languages tend to use more tokens, but after thing i realizend that languages with more expressiv syntax allow to write with less "words". But i also think it is a little bit like a race of watches. who realy wants to know what watch runs faster?

johnisgood 9mo ago

Concatenative languages like Factor and Forth are very token-efficient in theory. Theoretically optimal for raw lexical density. No parentheses, no commas, no argument delimiters, just whitespace-separated words, but stack shuffling can add overhead for complex data flow, unless you use "locals" in Factor, for example. C is surprisingly efficient as well. Minimal keywords, terse syntax, single-character operators. Not much boilerplate, and the core logic is dense. I think the worst languages are Java, C#, and Rust (lifetime annotations, verbose generics). In my opinion, C or Go for imperative code, Factor / Forth if the model knows them well.

Bootvis 9mo ago
I understand your logic but I found LLM's to be quite strong at C#. It makes little mistakes and the mistakes seem related to the complexity of what I'm doing, not the language itself.
- johnisgood 9mo ago
  See https://news.ycombinator.com/item?id=46586312 https://news.ycombinator.com/item?id=46586312. I cannot speak much for C#, but you may be right. Claude's Opus is really good.
Smaug123 9mo ago
Is that statement about C based on anything in particular? C was 18th of all the languages in the article's chart (the worst!), which I'd guess was due to the absence of a standard library.
- johnisgood 9mo ago
  Fair point. There is a distinction between syntactic efficiency (C is terse) and task-completion efficiency (what the benchmark likely measured). If the tasks involved string manipulation, hash maps, JSON, etc. then C pays a massive token tax because you are implementing what other languages provide in stdlib. Python has dict and json.loads(), C has malloc and strcmp. So: C tokenizes efficiently for equivalent logic, but stdlib poverty makes it expensive for typical benchmark tasks. Same applies to Factor/Forth, arguably worse.

aleph_minus_one 9mo ago

Relevant: On https://danuker.go.ro/programming-languages.html https://danuker.go.ro/programming-languages.html you can find charts of popularity (TIOBE) vs code density for various programming languages together with which programming languages are Pareto-optimal regarding these two criteria.

HtmlProgrammer 9mo ago

> I then told Claude Code to suggest a selection of the most popular programming languages If you’re going to write an article atleast do the base research yourself man

jwr 9mo ago

I program mostly in Clojure and I expected it to be near the top, as it tends to be very concise and expressive (qualities I really admire). I am getting excellent results from Claude Code (Opus 4.5), and I think this might be one of the reasons. I'm using Claude with a large code base and the token-efficiency of Clojure might help with fitting more into the context window.

kaliszad 9mo ago
I also program a lot in Clojure/Script. Do you also consider thinking token and the number of iterations in the token efficiency?
- jwr 9mo ago
  I don't think thinking tokens are affected, as LLMs "think" mostly in plain language, with occasional code snippets.
  - kaliszad 9mo ago
    I would assume for certain problems LLMs have a solution readily available for JavaScript/ TypeScript or similarly popular languages but not for Clojure/Script. Therefore my thinking was that the process of getting to a workable solution would be longer and more expensive in terms of tokens. I however don't have any relevant data on this so I may just be wrong.

nige123 9mo ago

Raku

torginus 9mo ago

This confirms my personal experience with switching to Go from C# - despite the e framework and language being MUCH simpler, the code usually ends up the same length. C# often has a 'nice' and 'performant' way of doing things (for example, strings are nice, but they allocate and are UTF16, but ReadOnlySpan<byte> is faster for UTF8, and can reuse buffers), the performant syntax often ends up being very verbose, with the nice syntax being barely shorter than Go's. Go also does the right thing by default, and its strings are basically array slices into UTF8 byte arrays.

neonsunset 9mo ago
[dead]

thw_9a83c 9mo ago

There is one class of languages missing in the comparison: Programming golf languages: E.g. Japt [1], Pyth [2] or Jelly [3]. Update: I noticed that the author mentions that "APL's famous terseness isn't a plus for LLMs." Isn't that just a design limitation of the LLM tokenizers? [1]: https://github.com/ETHproductions/japt https://github.com/ETHproductions/japt [2]: https://github.com/isaacg1/pyth https://github.com/isaacg1/pyth [3]: https://github.com/DennisMitchell/jellylanguage https://github.com/DennisMitchell/jellylanguage

tmtvl 9mo ago

I kinda (but not really because I don't much care about tokens and don't really know anything about models) wonder about Common Lisp. There's probably far fewer examples of CL code in any training sets than Clojure or Python or whatever, but it could still be somewhat interesting.

singularity2001 9mo ago

That's why I love Julia so much. Also it's semi-static?

eigenspace 9mo ago
Semantically, julia is a fully dynamic language. But the trick is that it does this by recognizing that being static is a constraint on a dynamic language, so it implements dynamic typing by stitching together islands of statically typed code.

6510 9mo ago

Curious about FORTH now

daft_pink 9mo ago

Wow, I’m really shocked that something like Javascript and Typescript which models tend to prefer wasn’t the most efficient.

zhisme 9mo ago

ruby. that's quite interesting, maybe because it reads mostly like plain english?

reena_signalhq 9mo ago

[dead]