13 ms·
Developed by Jordan Hubbard of NVIDIA (and FreeBSD). My understanding/experience is that LLM performance in a language scales with how well the language is rep
by thorum 9mo ago
Developed by Jordan Hubbard of NVIDIA (and FreeBSD).
My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data.
From that assumption, we might expect LLMs to actually do better with an existing language for which more training code is available, even if that language is more complex and seems like it should be “harder” to understand.
- whimsicalism 9mo agoeasy enough to solve with RL probably
- measurablefunc 9mo agoThere is no RL for programming languages. Especially ones w/ no significant amount of code.
- whimsicalism 9mo agonot even wrong
- measurablefunc 9mo agoExactly.
- nl 9mo agoI guess the op was implying that is something fixable fairly easily? (Which is true - it's easy to prompt your LLM with the language grammar, have it generate code and then RL on that) Easy in the sense of "it is only having enough GPUs to RL a coding capable LLM" anyway.
- measurablefunc 9mo agoIf you can generate code from the grammar then what exactly are you RLing? The point was to generate code in the first place so what does backpropagation get you here?
- nl 9mo agoPost RL you won't need to put the grammar in the prompt anymore.
- measurablefunc 9mo agoThe grammar of this language is no more than a few hundred tokens (thousands at worst) & current LLMs support context windows in the millions of tokens.
- nl 9mo agoSure. The point is that your statement about the ability to do RL is wrong. Additionally your response to the Deepseek paper in the other subthread shows profound and deliberate ignorance.
- measurablefunc 9mo agoTheorycrafting is very easy. Not a single person in this thread has shown any code to do what they're suggesting. You have access to the best models & yet you still haven't managed to prompt it to give you the code to prove your point so spare me any further theoretical responses. Either show the code to do exactly what you're saying is possible or admit you lack the relevant understanding to back up your claims.
- nl 9mo ago> You have access to the best models & yet you still haven't managed to prompt it to give you the code to prove your point so spare me any further theoretical responses. Either show the code to do exactly what you're saying is possible GPU poor here though... To quote someone (you...) on the internet: > More generally, don't ask random people on the internet to do work for you for free. https://news.ycombinator.com/item?id=46689232 https://news.ycombinator.com/item?id=46689232
- thorum 9mo agoGo read the DeepSeek R1 paper
- measurablefunc 9mo agoWhy would I do that? If you know something then quote the relevant passage & equation that says you can train code generators w/ RL on a novel language w/ little to no code to train on. More generally, don't ask random people on the internet to do work for you for free.
- thorum 9mo agoYour other comment sounded like you were interested in learning about how AI labs are applying RL to improve programming capability. If so, the DeepSeek R1 paper is a good introduction to the topic (maybe a bit out of date at this point, but very approachable). RL training works fine for low resource languages as long as you have tooling to verify outputs and enough compute to throw at the problem.
- whimsicalism 9mo agoimo generally not worth it to keep going when you encounter this sort of HN archetype
- measurablefunc 9mo agoSo you should have no problem bringing up the exact passages & equations they use for their policies.
- whimsicalism 9mo agowell, that’s one way to react to being provided with interesting reading material.
- measurablefunc 9mo agoBring up passage that supports your claim. I'll wait.
- nxobject 9mo agoI think it's depressingly true of any novel language/framework at this point, especially if they have novel ideas.
- vessenes 9mo agoA lot of this depends on your workflow. A language with great typing, type checking and good compiler errors will work better in a loop than one with a large surface overhead and syntax complexity, even if it's well represented. This is the instinct behind, e.g. https://github.com/toon-format/toon https://github.com/toon-format/toon, a json alternative format. They test LLM accuracy with the format against JSON, (and are generally slightly ahead of JSON). Additionally just the ability to put an entire language into context for an LLM - a single document explaining everything - is also likely to close the gap. I was skimming some nano files and while I can't say I loved how it looked, it did look extremely clear. Likely a benefit.
- btown 9mo agoThanks for sharing this! A question I've grappled with is "how do you make the DOM of a rendered webpage optimal for complex retrieval in both accuracy and tokens?" This could be a really useful transformation to throw in the mix!
- Zigurd 9mo agoIt's not just how well the language is represented. Obscure-ish APIs can trip up LLMs. I've been using Antigravity for a Flutter project that uses ATProto. Gemini is very strong at Dart coding, which makes picking up my 17th managed language a breeze. It's also very good at Flutter UI elements. It was noticeably less good at ATProto and its Dart API. The characteristics of failures have been interesting: As I anticipated it might be, an over ambitious refactoring was a train wreck, easily reverted. But something as simple as regenerating Android launcher icons in a Flutter project was a total blind spot. I had to Google that like some kind of naked savage running through the jungle.
- nl 9mo agoI have a vibe coded fantasy console. Getting Doom running on it was easy. Getting the Doom sound working on it involved me setting there typing "No I can't hear anything" over and over until it magically worked... Maybe I should have written a helper program to listen using the microphone or something.
- nxobject 9mo agoI was reading an Ars article by someone who'd hacked up Apple II and Atari 2600 emulators to provide state introspection/reproducible input via MCP, sockets, file I/O - would that work? https://github.com/benj-edwards/atari800-ai https://github.com/benj-edwards/atari800-ai https://github.com/benj-edwards/bobbin https://github.com/benj-edwards/bobbin
- nl 9mo agowow I love it.
- cmrdporcupine 9mo agoNot my experience, honestly. With a good code base for it to explore and good tooling, and a really good prompt I've had excellent results with frankly quite obscure things, including homegrown languages. As others said, the key is feedback and prompting. In a model with long context, it'll figure it out.
- rocha 9mo agoBut isn't this inefficient since the agent has to "bootstrap" its knowledge of the new language every time it's context window is reset?
- adastra22 9mo agoNo, it gets it “for free” just by looking around when it is figuring out how to solve whatever problem it is working on.
- vidarh 9mo agoYeah, I've had Claude work on my buggy, incomplete Ruby compiler written (mostly) in Ruby, which uses an s-expression like syntax with a custom "mini language" to implement low-level features that can't be done (or is impractical to do) in pure Ruby, and it only had minor problems with the s-expression language that was mostly fixed with a handful of lines in CLAUDE.md (and were, frankly, mostly my fault for making the language itself somewhat inconsistent) and e.g. when it write a bigint implementation, I had to "tell it off" for too readily resorting to the s-expression syntax since it seemed to "prefer it" over writing high-level code in Ruby.
- cmrdporcupine 9mo agoEven 3 years ago, GH Copilot, hardly the most intelligent of LLMs was suggesting/writing bytecode in my custom VM, writing full programs in bytecode for a custom VM just by looking at a couple examples. That's when I smelled that things were getting a little crazy.
- adastra22 9mo agoI don’t think that assumption holds. For example, only recently have agents started getting Rust code right on the first try, but that hasn’t mattered in the past because the rust compiler and linters give such good feedback that it immediately fixes whatever goof it made. This does fill up context a little faster, (1) not as much as debugging the problem would have in a dynamic language, and (2) better agentic frameworks are coming that “rewrite” context history for dynamic on the fly context compression.
- root_axis 9mo ago> that hasn’t mattered in the past because the rust compiler and linters give such good feedback that it immediately fixes whatever goof it made. This isn't even true today. Source: heavy user of claude code and gemini with rust for almost 2 years now.
- PunchyHamster 9mo agoso you're saying... the assumption actually holds
- adastra22 9mo agoNo, it’s the exact opposite of the assumption. It doesn’t matter how represented the language is in the training data, so long as the surrounding infrastructure is good.
- bevr1337 9mo ago> because the rust compiler and linters give such good feedback that it immediately fixes whatever goof it made. I still experience agents slipping in a `todo!` and other hacks to get code to compile, lint, and pass tests. The loop with tests and doc tests are really nice, agreed, but it'll still shit out bad code.
- NewsaHackO 9mo agoI wonder if there is a way to create a sort of 'transpilation' layer to a new language like this for existing languages, so that it would be able to use all of the available training from other languages. Something that's like AST to AST. Though I wonder if it would only work in the initial training or fine-tuning stage.
- nl 9mo ago> My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data. This isn't really true. LLMs understand grammars really really well. If you have a grammar for your language the LLM can one-shot perfect code. What they don't know is the tooling around the language. But again, this is pretty easily fixed - they are good at exploring cli tools.
- nemo1618 9mo agoBlackpill is that, for this reason, the mainstream languages we have today will be the final (human-designed) languages to be relevant on a global scale. Eventually AIs will create their own languages. And humans will, of course, continue designing hobbyist languages for fun. But in terms of influence, there will not be another human language that takes the programming world by storm. There simply is not enough time left.
- deleted 9mo ago[deleted]
- rzmmm 9mo agoMy impression is that AI models need large amounts of quality training data. "Data contamination", i.e. AI output in the training data set has been a problem for years.
- vidarh 9mo agoI mostly agree, and I think a combination of good representation and tooling that lets it self-correct quickly will do better than new language in the short term. In the long term I expect it won't matter - already GPT3.5 was able to reason about the basic semantics of programs in languages "synthesised" zero-shot in context by just describing it as a combination of existing languages (e.g. "Ruby with INTERCAL's COME FROM") or by providing a grammar (e.g. simple EBNF plus some notes on new/different constructs) reasonably well and could explain what a program written in a franken-language it had not seen before was likely to do. I think long before there is enough training data for a new language to be on equal grounds in that respect, we should expect the models to be good enough at this that you could just provide a terse language spec. But at the same time, I'd expect the same improvement to future models to be good enough at working with existing languages that it's pointless to tailor languages to LLMs.
- boxed 9mo agoClaude is very good with Elm, which there should be quite little training data.