5 ms·

I don't think it is correct to call this a programming language. This program enforces some structure on your specifications for LLMs and provides some guardra

I don't think it is correct to call this a programming language. This program enforces some structure on your specifications for LLMs and provides some guardrails, which is absolutely a move in the right direction, but these are related more to formatting rather than specification, i.e. it provides syntax without transparent or unambiguous semantics. Ultimately this is more of a markup format than a programming language. Since: > The next section is the description of the function. Here you explain what the function should do. Being more explicit here will reduce variability in the generated output and improve reliability in behavior, but it's up to you just how explicit you will be and how much you leave to the LLM to figure out. There are reasons that natural language is problematic when specifying systems! I wish I could tape a sign that says this to the top of HN given the number of projects on the front page the past few months calling all sorts of things compilers that are just thin wrappers over the OpenAI API. These projects frame programming with natural language as though it is something desirable, like it is the holy grail of computer science that will open up engineering to the masses. But we use programming languages not to make programming difficult, but because unrestricted natural language is problematic. Systems are best specified with highly constrained languages that are (more) semantically unambiguous. Without sufficient constraints, there's a good chance that we don't even know what we ourselves are talking about when we specify systems -- how can we hope then that computers will? Even software engineers misinterpret and misunderstand requirements, requirements are poorly specified, or requirements are (apparently) well-understood but the space of possible inputs and use cases are not. This is why mathematicians use a (loosely) restricted jargon and notation, and even they run into difficulties All of that said: LLMs are surely a useful tool for software engineering and systems-building -- I personally use them most days to that end. But make no mistake that this is a markup language with some added guardrails to help users make better specifications for LLMs and verify the output. It is not a programming language, and programming with natural language is not generally something that is generally possible or even desirable

Der_Einzige 3y ago
If it's turing complete, it's a programming language. That's it. There's no other requirements.
- williamstein 3y ago
  Turing complete is a property that a programming language may or may not have. Bloop is a non Turing complete programming language: https://en.m.wikipedia.org/wiki/BlooP_and_FlooP https://en.m.wikipedia.org/wiki/BlooP_and_FlooP
- mepian 3y ago
  Let me introduce you to https://en.wikipedia.org/wiki/Total_functional_programming https://en.wikipedia.org/wiki/Total_functional_programming
- jachee 3y ago
  So, Magic: the Gathering (being Turing complete[0]) is a programming language? [0]: https://arxiv.org/abs/1904.09828 https://arxiv.org/abs/1904.09828
FireInsight 3y ago
> calling all sorts of things compilers that are thin wrappers over the OpenAI API Well compilation and transpilation are kind or related, and translation is kind of an ambigous version of transpilation, and LLMs are kind of a more nondeterministic superset of natural language translation (processing), which really makes calling it a compiler a huge stretch, I agree. This all makes me wonder, could it be possible for an LLM to spit out the bits, or even assembly, for a hello world program written in another programming language. That'd kind of make it a really bad compiler.
ISV_Damocles 3y ago
I don't believe that I can change your mind on this, so I didn't intend to respond, but as this is the top comment, I do want to provide a rebuttal on why we do think this is actually a programming language, that the code we have written is actually a compiler, and why Marsha is a useful exploration of the programming language design space. First, a programming language is just a syntax to describe functionality that could be turned into an actual program. Lisp[1] was defined in 1958 but didn't have a full compiler until 1962. Was it not a programming language in the intervening 4 years? Marsha does not fall into this, since it can already generate working code, but the bar for what is a programming language, I believe, is lower than most would immediately think. Second, a programming language does not need to be imperative to be a programming language, or languages like Lean[2] that have you write proofs that the compiler then figures out how to generate the code to fulfill would not be programming languages. Lean, Coq, and other such languages are much more technically impressive than Marsha, true, but they share the property you describe the properties a function should have and then the compiler generates the program that fulfills those properties. Marsha differs from these Proof-based languages in that poor specificity still produces some sort of program instead of a compilation error, which makes it sort of like Javascript that will do something with the code you write as long as it is syntactically valid. This is not a desirable property of Marsha, but it is a trade-off that in practice makes it more immediately usable to a larger number of people than Lean or Coq, because the skill level required is lower. This is also, as you allude to, the current state of the world in most software development -- project managers come up with high-level requirements for new features, technical leads on engineering teams convert this into tasks and requirements for individual contributors who then write the code and tests which are then peer reviewed by the team as a sanity check and then committed. This process may or may not cover all situations and the specifications at all levels are likely not as rigorous as what Lean would require of you. Marsha mimics this process, starting from the tech lead level and bleeding into the individual contributor level. The type and function descriptions are analogous to the tech lead requirements and the examples are analogous to the test suite the individual contributor would write. Just like in real world development, if these are not well specified, the resulting code will likely have logic bugs that would need to be addressed with a stricter definition and improved test cases. The compiler consumes this definition into an AST[3], walks the tree to generate intermediate forms, and generates an output in a format that can be executed by a computer. Some use "transpiler" for a compiler that targets another language, but that is a subset of compilers, not a separate kind of tool, in my opinion, or the Java compiler would be a "transpiler" for the JVM bytecode format that is also not directly executable by a computer. We are still in the very early stages with Marsha and agree that more syntax could be helpful -- we already have 4 different syntactic components to Marsha versus the fully open-ended text entry behavior of Github Copilot or ChatGPT. But what makes Marsha interesting (to me) is that it makes it possible to explore a totally new dimension in programming language design: the formalization of the syntax to define a program itself. In many papers on new algorithms, the logic is often described in a human-readable list of steps without the hard specificity of programming languages, improving the ability of the reader to understand the core of the algorithm, rather than getting bogged down in the implementation details of this or that programming language. There is still a formalism, but it differs from that of traditional programming languages, and Marsha lets you work with your computer in a similar way. Are there cases where this is a bad idea? Absolutely. Just like there are cases where writing your code in Python is a bad idea versus writing it in Rust. There is no perfect programming language useful for all scenarios, and probably never will exist. But there will be a subset of situations where the trade-offs Marsha provides makes sense. By being more forgiving than even the most forgiving interpreted languages out there, Marsha is in a good position to fill that niche if the primary barrier is difficulty. [1]: https://en.wikipedia.org/wiki/Lisp_(programming_language)#History https://en.wikipedia.org/wiki/Lisp_(programming_language)#Hi... [2]: https://en.wikipedia.org/wiki/Lean_(proof_assistant) https://en.wikipedia.org/wiki/Lean_(proof_assistant) [3]: https://github.com/alantech/marsha/blob/main/marsha/parse.py#L77 https://github.com/alantech/marsha/blob/main/marsha/parse.py...
obeavs 3y ago
Nicely considered. I've spent the last two months writing a paper on how natural language (and the ambiguities within) drive most of the inefficiency in today's financial markets. Properly designed DSLs for these spaces (which we've been working on) would do absolute wonders for how anything from housing development to critical infrastructure is financed. It's remarkable how much more clearly logic is when semantic flourish (and associated ambiguity) from the English language is thoughtfully constrained away, and it's always seemed interesting to see how computer engineers see natural language as the target. phosphor.co/20x if you'd like to see how this works in contract law and finance :).
- ibestvina 3y ago
  I'm always drawn to these types of initiatives, and your whitepaper looks (at least with my very limited knowledge of the domain) interesting. What I am always wondering, and maybe you can give some details here, is the following: isn't the fact that regulations are in natural language, with all its ambiguity, a necessary requirement to have the system operate without being fully specified? In other words, wouldn't any kind of strict DSL force us to think through all the edge cases that might possibly arise, instead of dealing with them when they do arise, which is basically what the judiciary is for? And isn't that a price too high to implement these kinds of systems?

Der_Einzige 3y ago

If it's turing complete, it's a programming language. That's it. There's no other requirements.

williamstein 3y ago
Turing complete is a property that a programming language may or may not have. Bloop is a non Turing complete programming language: https://en.m.wikipedia.org/wiki/BlooP_and_FlooP https://en.m.wikipedia.org/wiki/BlooP_and_FlooP
mepian 3y ago
Let me introduce you to https://en.wikipedia.org/wiki/Total_functional_programming https://en.wikipedia.org/wiki/Total_functional_programming
jachee 3y ago
So, Magic: the Gathering (being Turing complete[0]) is a programming language? [0]: https://arxiv.org/abs/1904.09828 https://arxiv.org/abs/1904.09828

FireInsight 3y ago

> calling all sorts of things compilers that are thin wrappers over the OpenAI API Well compilation and transpilation are kind or related, and translation is kind of an ambigous version of transpilation, and LLMs are kind of a more nondeterministic superset of natural language translation (processing), which really makes calling it a compiler a huge stretch, I agree. This all makes me wonder, could it be possible for an LLM to spit out the bits, or even assembly, for a hello world program written in another programming language. That'd kind of make it a really bad compiler.

ISV_Damocles 3y ago

I don't believe that I can change your mind on this, so I didn't intend to respond, but as this is the top comment, I do want to provide a rebuttal on why we do think this is actually a programming language, that the code we have written is actually a compiler, and why Marsha is a useful exploration of the programming language design space. First, a programming language is just a syntax to describe functionality that could be turned into an actual program. Lisp[1] was defined in 1958 but didn't have a full compiler until 1962. Was it not a programming language in the intervening 4 years? Marsha does not fall into this, since it can already generate working code, but the bar for what is a programming language, I believe, is lower than most would immediately think. Second, a programming language does not need to be imperative to be a programming language, or languages like Lean[2] that have you write proofs that the compiler then figures out how to generate the code to fulfill would not be programming languages. Lean, Coq, and other such languages are much more technically impressive than Marsha, true, but they share the property you describe the properties a function should have and then the compiler generates the program that fulfills those properties. Marsha differs from these Proof-based languages in that poor specificity still produces some sort of program instead of a compilation error, which makes it sort of like Javascript that will do something with the code you write as long as it is syntactically valid. This is not a desirable property of Marsha, but it is a trade-off that in practice makes it more immediately usable to a larger number of people than Lean or Coq, because the skill level required is lower. This is also, as you allude to, the current state of the world in most software development -- project managers come up with high-level requirements for new features, technical leads on engineering teams convert this into tasks and requirements for individual contributors who then write the code and tests which are then peer reviewed by the team as a sanity check and then committed. This process may or may not cover all situations and the specifications at all levels are likely not as rigorous as what Lean would require of you. Marsha mimics this process, starting from the tech lead level and bleeding into the individual contributor level. The type and function descriptions are analogous to the tech lead requirements and the examples are analogous to the test suite the individual contributor would write. Just like in real world development, if these are not well specified, the resulting code will likely have logic bugs that would need to be addressed with a stricter definition and improved test cases. The compiler consumes this definition into an AST[3], walks the tree to generate intermediate forms, and generates an output in a format that can be executed by a computer. Some use "transpiler" for a compiler that targets another language, but that is a subset of compilers, not a separate kind of tool, in my opinion, or the Java compiler would be a "transpiler" for the JVM bytecode format that is also not directly executable by a computer. We are still in the very early stages with Marsha and agree that more syntax could be helpful -- we already have 4 different syntactic components to Marsha versus the fully open-ended text entry behavior of Github Copilot or ChatGPT. But what makes Marsha interesting (to me) is that it makes it possible to explore a totally new dimension in programming language design: the formalization of the syntax to define a program itself. In many papers on new algorithms, the logic is often described in a human-readable list of steps without the hard specificity of programming languages, improving the ability of the reader to understand the core of the algorithm, rather than getting bogged down in the implementation details of this or that programming language. There is still a formalism, but it differs from that of traditional programming languages, and Marsha lets you work with your computer in a similar way. Are there cases where this is a bad idea? Absolutely. Just like there are cases where writing your code in Python is a bad idea versus writing it in Rust. There is no perfect programming language useful for all scenarios, and probably never will exist. But there will be a subset of situations where the trade-offs Marsha provides makes sense. By being more forgiving than even the most forgiving interpreted languages out there, Marsha is in a good position to fill that niche if the primary barrier is difficulty. [1]: https://en.wikipedia.org/wiki/Lisp_(programming_language)#History https://en.wikipedia.org/wiki/Lisp_(programming_language)#Hi... [2]: https://en.wikipedia.org/wiki/Lean_(proof_assistant) https://en.wikipedia.org/wiki/Lean_(proof_assistant) [3]: https://github.com/alantech/marsha/blob/main/marsha/parse.py#L77 https://github.com/alantech/marsha/blob/main/marsha/parse.py...

obeavs 3y ago

Nicely considered. I've spent the last two months writing a paper on how natural language (and the ambiguities within) drive most of the inefficiency in today's financial markets. Properly designed DSLs for these spaces (which we've been working on) would do absolute wonders for how anything from housing development to critical infrastructure is financed. It's remarkable how much more clearly logic is when semantic flourish (and associated ambiguity) from the English language is thoughtfully constrained away, and it's always seemed interesting to see how computer engineers see natural language as the target. phosphor.co/20x if you'd like to see how this works in contract law and finance :).

ibestvina 3y ago
I'm always drawn to these types of initiatives, and your whitepaper looks (at least with my very limited knowledge of the domain) interesting. What I am always wondering, and maybe you can give some details here, is the following: isn't the fact that regulations are in natural language, with all its ambiguity, a necessary requirement to have the system operate without being fully specified? In other words, wouldn't any kind of strict DSL force us to think through all the edge cases that might possibly arise, instead of dealing with them when they do arise, which is basically what the judiciary is for? And isn't that a price too high to implement these kinds of systems?