3 ms·
This would have been interesting about a decade ago, but today AIs all know SQL, and I haven't written it myself in a while. Since it seems like the quantity o
by a2ff6eeb0 2mo ago
This would have been interesting about a decade ago, but today AIs all know SQL, and I haven't written it myself in a while.
Since it seems like the quantity of training data dominates AI performance, and AI doesn't yet internalize experience with new tools, it seems like a bad idea to stray from the training set.
Without repeatable benchmarks, it feels like obsessing over a language's syntax and semantics feels a little like debating whether you write assembly using AT&T or Intel syntax.
- ModernMech 2mo agoAIs would benefit from better query languages for the same reasons people would.
- a2ff6eeb0 2mo agoThe difference is that the bulk of what an AI knows is baked in when it's trained, at least for now. There's no way for it to learn a language and improve with it.
- ModernMech 2mo agoThat's not quite true in my experience; AI can pick up new languages very quickly and are able to adapt to novel syntax and semantics with just a description and a few examples. What's also baked into the AI are decades of PL research and it can quickly deploy esoteric PL concepts not found in 99% of languages. In my experience it's rather people who have the most trouble with new languages, as the difference between the PL frontier and languages that most people use is quite extreme. Conversely, AI is adept at staking out a point in the PL design space and developing a grammar and vocabulary around it. Then it writes a parser and interpreter to execute whatever semantics, writes a standard library to support writing programs, and finally writes the compiler in itself. Because it's so good at doing this you can do a lot of exploration whereas before it would take years now it takes months.
- a2ff6eeb0 2mo agoIt'll do something, but with a higher error rate and far more iterations needed.
- ModernMech 2mo agoAgain it's just not been my experience through testing so I'm curious what kind of measurements you're citing here.
- a2ff6eeb0 2mo agoHere's some related work that matches my observations: https://danluu.com/pl-tokens/ https://danluu.com/pl-tokens/
- ModernMech 2mo agoThanks, yeah I remember when that made the rounds a couple weeks ago. Although what I'm proposing is a little different than what's covered there: the AI designing a language for a particular task, writing the runtime to implement the language, and then solving the task in the language it designed. The blog rather is about how an AI performs with languages designed by people for general purposes.
- a2ff6eeb0 2mo agoWhat evals did you use to compare that to using an existing language with a large corpus of training data?
- ModernMech 2mo agoI've got two experiments running now and I'm going to do a third soon. The first area is linear algebra, where there's a readily available notation for the AI to operationalize. In this area it's very easy for the AI to one-shot write correct algorithms that are shorter than typical languages because they are already written down, and the notation is very compact, so all it has to do is a direct translation from a textbook. I'm now working on comparing with algorithms it doesn't already know. The second area I'm working on now doesn't have a readily available notation, which is state machines. Here, the AI can concoct a very terse state machine representation and write very complex state machines that can be statically analyzed so it has a better time than writing in a plain language without that capability. Now I'm trying to test how it fares against other state machine DSLs. The next area I will move to after this I think is music, which also has a readily available notation that AI can operationalize. No numbers to report yet but I'll publish my research when it's done.