4 ms·

An AI-first program synthesis framework built around a new programming language

112 points by tosh 1y ago

sys13 1y ago
Glad to see focus being put on keeping humans in the drivers seat, democratizing coding with the help of AI. The syntax is probably still too verbose to be easily accessible, but I like the overall approach.
AIPedant 1y ago
I know ACM Queue is a non-peer-reviewed magazine for practitioners but this still feels like too much of an advertisement, without any attempt whatsoever to discuss downsides or limitations. This really doesn't inspire confidence: While this may seem like a whimsical example, it is not intrinsically easier or harder for an AI model compared to solving a real-world problem from a human perspective. The model processes both simple and complex problems using the same underlying mechanism. To lessen the cognitive load for the human reader, however, we will stick to simple targeted examples in this article. For LLMs this is blatantly false - in fact asking about "used textbooks" instead of "apples" is measurably more likely to result in an error! Maybe the (deterministic, Prolog-style) Universalis language mitigates this. But since Automind (an LLM, I think) is responsible for pre/post validation, naively I would expect it to sometimes output incorrect Universalis code and incorrectly claim an assertion holds when it does not. Maybe I am making a mountain out of a molehill but this bit about "lessen the cognitive load of the human reader" is kind of obnoxious. Show me how this handles a slightly nontrivial problem, don't assume I'm too stupid to understand it by trying to impress me with the happy path.
- tannhaeuser 1y ago
  Prolog works indeed very well as target for generation by an LLM, for input problems limited and similar enough in nature to given classes of templated in-context examples, so well indeed that the lack of a succinct, exhaustive text description of your problem is becoming the issue. At which point you can specify your problem in Prolog directly considering Prolog was also invented to model natural language parsing and not just for solving constraint/logic problems, or could employ ILP techniques to learn or optimize Prolog solvers from existing problem solutions rather than text descriptions. See [1]. [1]: https://quantumprolog.sgml.net https://quantumprolog.sgml.net
  - YeGoblynQueenne 1y ago
    >> Since we're not doing original research, but rather intend to demonstrate a port of the Aleph ILP package to ISO Prolog running on Quantum Prolog, we cite the problem's definition in full from the original paper (ilp09): Aleph? In 2025. That's just lazy, now. At the very least they should try Metagol or Popper, both with dozens of recent publications (and I'm not even promoting my own work).
    - tannhaeuser 1y ago
      You're not wrong, and alternatives were considered, but those were really not fit to be ported to ISO Prolog in bounded time with complete lack of tests or even a basic reproducible demo, uncontrolled usage of non-ISO libs and features only available on the originally targetted Prolog implementation, and other issues typical of "academic codes."
      - YeGoblynQueenne 1y ago
        The lack of unit tests is something I'm guilty of too and you're very right about it. The community is dimly aware of the fact that systems are more on the "academic prototype" side of things and less on the "enterprise software" but there's so little interest from industry that nobody is going to spend significant effort to change that. Kind of a catch 22 maybe. How about ISO? Why was this a requirement, out of curiosity?
        tannhaeuser 1y ago
        Quantum Prolog is pure ISO and was the primary target for porting here. The idea is really just putting effort behind libs that can be shared by other non-legacy Prologs and thus get more people involved.
        YeGoblynQueenne 1y ago
        Look, I'll be blunt. If ISO is keeping you from running any ILP system newer than Aleph and you have a good reason to want to use an ILP system then you should reconsider the benefits of ISO. There's no ISO for LLMs for example, yet you'll have no trouble at all convincing people to get involved, there. That said, I think Vanilla should be easy to ISO-fy. It's just a meta-interpreter really. It uses tabling and I'm guessing that's not ISO (it's more like the Wild West) but it is not indispensible. https://github.com/stassa/vanilla/blob/master/src/vanilla.pl https://github.com/stassa/vanilla/blob/master/src/vanilla.pl I could have a look, but I have no idea what's in the ISO standard (and that's another bugbear- the ISO docs are not free. What?). I guess I could try to run it on your engine and see what the errors say though. P.S. Re: "academic codes" I got documentation, logging, and custom errors and that's more than you can say for 100% of academic prototypes plus ~80% of industry software also. I do have unit tests, even, just not many.
duskwuff 1y ago
This article feels extremely imprecise. The syntax of the "language" changes from example to example, control structures like conditionals are expressed in English prose, some examples are solved by "do all the work for me" functions like the "toPdf()" example... This whole thing feels like an elaborate LLM fantasy. Is there any real, usable language behind these examples, or is the author just role-playing with ChatGPT?
- YeGoblynQueenne 1y ago
  More to the point, is ChatGPT role-playing the author?
  - mannycalavera42 1y ago
    lol
ethanwillis 1y ago
> Universalis ensures that even those with minimal experience in programming can perform advanced data manipulations. Is it a good thing to make this easier? We're drowning in garbage already.
rvasa 1y ago
Great to start off ... then we will end-up reinventing/re-specifying functions for reusability, module/packages for higher-level grouping, types/classes, state machines, control-flows [with the nuances for edge cases and exit conditions], then we will need error control, exceptions; sooner or later concurrency, parallelism, data structures, recursion [lets throw in monads for the Haskellians amoung us]; who knows .. we may even end up with GOTOs peppered all over the English sentences [with global labels] & wake up to the scoping, parameter passing. We can have a whole lot of new fights if we need object-oriented programming; figure out new design patterns with special "Token Factory Factories". We took a few decades to figure out how to specify & evolve current code to solve a certain class of problems [nothing is perfect .. but it seems to work at scale with trade-offs]. Shall watch this from a distance with pop-corn.
pjmlp 1y ago
So after Haskell, and helping with LINQ, this is what Erik Meijer has been focusing on.
gavinray 1y ago
> "The actual implementation of Universalis uses Kotlin DataFrames" Neat. My favorite talk at the 2025 Kotlin Conf was the demo of the new compiler plugin that allows ad-hoc type inference from DataFrame expressions. Essentially, say that you have an input type like: type User { name: String, age: Int } And you do something like ".map(it => { doubledAge: it.age * 2 })" The inferred type of that intermediate operation is now: type User_synthetic { doubledAge: Int } Which is wild, since you essentially have TypeScript-like inference in a JVM language. Timestamp to talk: https://www.youtube.com/watch?v=F5NaqGF9oT4&t=543s https://www.youtube.com/watch?v=F5NaqGF9oT4&t=543s

112 points by tosh 1y ago

Glad to see focus being put on keeping humans in the drivers seat, democratizing coding with the help of AI. The syntax is probably still too verbose to be easily accessible, but I like the overall approach.

AIPedant 1y ago

I know ACM Queue is a non-peer-reviewed magazine for practitioners but this still feels like too much of an advertisement, without any attempt whatsoever to discuss downsides or limitations. This really doesn't inspire confidence: While this may seem like a whimsical example, it is not intrinsically easier or harder for an AI model compared to solving a real-world problem from a human perspective. The model processes both simple and complex problems using the same underlying mechanism. To lessen the cognitive load for the human reader, however, we will stick to simple targeted examples in this article. For LLMs this is blatantly false - in fact asking about "used textbooks" instead of "apples" is measurably more likely to result in an error! Maybe the (deterministic, Prolog-style) Universalis language mitigates this. But since Automind (an LLM, I think) is responsible for pre/post validation, naively I would expect it to sometimes output incorrect Universalis code and incorrectly claim an assertion holds when it does not. Maybe I am making a mountain out of a molehill but this bit about "lessen the cognitive load of the human reader" is kind of obnoxious. Show me how this handles a slightly nontrivial problem, don't assume I'm too stupid to understand it by trying to impress me with the happy path.

tannhaeuser 1y ago
Prolog works indeed very well as target for generation by an LLM, for input problems limited and similar enough in nature to given classes of templated in-context examples, so well indeed that the lack of a succinct, exhaustive text description of your problem is becoming the issue. At which point you can specify your problem in Prolog directly considering Prolog was also invented to model natural language parsing and not just for solving constraint/logic problems, or could employ ILP techniques to learn or optimize Prolog solvers from existing problem solutions rather than text descriptions. See [1]. [1]: https://quantumprolog.sgml.net https://quantumprolog.sgml.net
- YeGoblynQueenne 1y ago
  >> Since we're not doing original research, but rather intend to demonstrate a port of the Aleph ILP package to ISO Prolog running on Quantum Prolog, we cite the problem's definition in full from the original paper (ilp09): Aleph? In 2025. That's just lazy, now. At the very least they should try Metagol or Popper, both with dozens of recent publications (and I'm not even promoting my own work).
  - tannhaeuser 1y ago
    You're not wrong, and alternatives were considered, but those were really not fit to be ported to ISO Prolog in bounded time with complete lack of tests or even a basic reproducible demo, uncontrolled usage of non-ISO libs and features only available on the originally targetted Prolog implementation, and other issues typical of "academic codes."
    - YeGoblynQueenne 1y ago
      The lack of unit tests is something I'm guilty of too and you're very right about it. The community is dimly aware of the fact that systems are more on the "academic prototype" side of things and less on the "enterprise software" but there's so little interest from industry that nobody is going to spend significant effort to change that. Kind of a catch 22 maybe. How about ISO? Why was this a requirement, out of curiosity?
      - tannhaeuser 1y ago
        Quantum Prolog is pure ISO and was the primary target for porting here. The idea is really just putting effort behind libs that can be shared by other non-legacy Prologs and thus get more people involved.
        YeGoblynQueenne 1y ago
        Look, I'll be blunt. If ISO is keeping you from running any ILP system newer than Aleph and you have a good reason to want to use an ILP system then you should reconsider the benefits of ISO. There's no ISO for LLMs for example, yet you'll have no trouble at all convincing people to get involved, there. That said, I think Vanilla should be easy to ISO-fy. It's just a meta-interpreter really. It uses tabling and I'm guessing that's not ISO (it's more like the Wild West) but it is not indispensible. https://github.com/stassa/vanilla/blob/master/src/vanilla.pl https://github.com/stassa/vanilla/blob/master/src/vanilla.pl I could have a look, but I have no idea what's in the ISO standard (and that's another bugbear- the ISO docs are not free. What?). I guess I could try to run it on your engine and see what the errors say though. P.S. Re: "academic codes" I got documentation, logging, and custom errors and that's more than you can say for 100% of academic prototypes plus ~80% of industry software also. I do have unit tests, even, just not many.

duskwuff 1y ago

This article feels extremely imprecise. The syntax of the "language" changes from example to example, control structures like conditionals are expressed in English prose, some examples are solved by "do all the work for me" functions like the "toPdf()" example... This whole thing feels like an elaborate LLM fantasy. Is there any real, usable language behind these examples, or is the author just role-playing with ChatGPT?

YeGoblynQueenne 1y ago
More to the point, is ChatGPT role-playing the author?
- mannycalavera42 1y ago
  lol

ethanwillis 1y ago

> Universalis ensures that even those with minimal experience in programming can perform advanced data manipulations. Is it a good thing to make this easier? We're drowning in garbage already.

rvasa 1y ago

Great to start off ... then we will end-up reinventing/re-specifying functions for reusability, module/packages for higher-level grouping, types/classes, state machines, control-flows [with the nuances for edge cases and exit conditions], then we will need error control, exceptions; sooner or later concurrency, parallelism, data structures, recursion [lets throw in monads for the Haskellians amoung us]; who knows .. we may even end up with GOTOs peppered all over the English sentences [with global labels] & wake up to the scoping, parameter passing. We can have a whole lot of new fights if we need object-oriented programming; figure out new design patterns with special "Token Factory Factories". We took a few decades to figure out how to specify & evolve current code to solve a certain class of problems [nothing is perfect .. but it seems to work at scale with trade-offs]. Shall watch this from a distance with pop-corn.

pjmlp 1y ago

So after Haskell, and helping with LINQ, this is what Erik Meijer has been focusing on.

gavinray 1y ago

> "The actual implementation of Universalis uses Kotlin DataFrames" Neat. My favorite talk at the 2025 Kotlin Conf was the demo of the new compiler plugin that allows ad-hoc type inference from DataFrame expressions. Essentially, say that you have an input type like: type User { name: String, age: Int } And you do something like ".map(it => { doubledAge: it.age * 2 })" The inferred type of that intermediate operation is now: type User_synthetic { doubledAge: Int } Which is wild, since you essentially have TypeScript-like inference in a JVM language. Timestamp to talk: https://www.youtube.com/watch?v=F5NaqGF9oT4&t=543s https://www.youtube.com/watch?v=F5NaqGF9oT4&t=543s