9 ms·

Show HN: Marsha – An LLM-Based Programming Language

140 points by ISV_Damocles 3y ago

ilaksh 3y ago
This is great. Have you thought about adding tree of thoughts?
- pombo 3y ago
  one of the contributors here. we have a statistical test script we can run on our branches to test the compile time, cost and reliability. we want to try tree of thought and also something like https://www.reddit.com/r/ChatGPT/comments/14d7pfz/become_god_like_prompt_engineer_with_this_one/ https://www.reddit.com/r/ChatGPT/comments/14d7pfz/become_god.... that said, we found that when we asked GPT to first explain why a test case is failing and then to correct that failure instead of just asking it to correct the failure, unexpectedly costs went up and reliability went down
  - ISV_Damocles 3y ago
    Btw, here's the test job: https://github.com/alantech/marsha/blob/main/.github/workflows/time.yaml https://github.com/alantech/marsha/blob/main/.github/workflo... And the core script for the job: https://github.com/alantech/marsha/blob/main/marsha/.time.py https://github.com/alantech/marsha/blob/main/marsha/.time.py
franciscomello 3y ago
Sounds very interesting.
Winse 3y ago
I thought this would kind of just be a pile of garbage, but I have to admit I was drawn in. There are some interesting novel pieces in Marsha and I am somewhat impressed with this project.
- ISV_Damocles 3y ago
  yeah, the AI hype cycle can be a bit tiring, but I'm glad you took a second look. What parts did you find interesting, in particular?
- weego 3y ago
  The most interesting thing for me is them providing examples of the function made me realise this is the logical conclusion of TDD. Write tests to build the input -> output contracts, have AI build the logic that conforms to those contracts.
jstarfish 3y ago
Heh...and here I thought I was being clever using Inform 7 as a metalanguage. This is way more concise.
- ISV_Damocles 3y ago
  Well, to be fair, we do bulk up what you type quite a bit[1] to improve GPT's response. [1]: https://github.com/alantech/marsha/blob/main/marsha/parse.py#L114-L138 https://github.com/alantech/marsha/blob/main/marsha/parse.py...
cabalamat 3y ago
Why not just define the function headers in Python? It's less verbose.
- ISV_Damocles 3y ago
  Simply, Marsha is not Python. ;) More seriously, in Python, types are optional and the exact behavior is explicitly defined at runtime, but the author of the code already had a behavior in mind when they wrote the code. Getting the intended type information from the user to provide to the LLM improves the quality of the generated code, which is actually somewhat lossy. The "intended type information" isn't "explicit type information", though. We support any character besides these four as part of a type `(),:`. That includes spaces and periods, so you can describe the type as a short sentence if you aren't exactly sure what the structure ought to be.[1] And since in Marsha you don't actually write an imperative function body, argument names don't matter, only the argument order (as Marsha does not support named parameters, at least right now) so there's no argument naming in the declaration, which can make it more succinct than Python def lines in certain situations, though to be fair none of our examples do this. [1]: https://github.com/alantech/marsha/blob/main/examples/web/duckduckgo.mrsh#L1 https://github.com/alantech/marsha/blob/main/examples/web/du...
cabalamat 3y ago
How easy would it be to make it output programs in a functional language, such as Haskell? it would be nice if it could be made to produce functions that are guaranteed to be free from side effects. In the long term I can imagine that the output languages of code generators like this might be ones specially designed to work well with AI code generators.
- ISV_Damocles 3y ago
  So, targeting other languages is on our roadmap, and we have found higher reliability when we stick to a functional style with Marsha, so I think Haskell will be one of the "good ones" amongst target languages, eventually. But we are focusing on Python first because it's one of the most popular languages on the planet and many LLMs out there are able to generate high quality code for it, while less-used languages tend to produce worse results. Haskell may or may not need a manually-tuned LLM to generate solid results.
  - cabalamat 3y ago
    > I think Haskell will be one of the "good ones" amongst target languages, eventually Another advantage of Haskell is that its side-effect-free nature makes it harder for a rogue AI to put in malicious code, thus helping to solve the alignment problem. It may be that languages and OSes will be specifically designed to put everything in a sandbox (or multiple layers of sandboxing), which will also be helpful when humans write code. > while less-used languages tend to produce worse results I figured that might be the case. > Haskell may or may not need a manually-tuned LLM to generate solid results. One possibility would be to write some Haskell programs, that illustrate paradigms in the language and train the LLM on them. Then get the LLM to solve similar problems, and automatically test how good its solutions are on grounds of solving the problem (passing test cases), performance (time and space used) and how easy it is for humans to understand the programs. Then use those solution to train the LLM to be better at Haskell, and give it another set of slightly harder problems to solve. In this way it would learn programming pretty much as humans do, by solving incrementally harder problems.
_ea1k 3y ago
This feels inevitable to me. All software engineering problems can be solved by the addition of another layer of abstraction. Why not abstract away the "how" layer and focus completely on the "what" layer? For a lot of data processing and integration problems, this would both eliminate a lot of work and increase reliability.
- ModernMech 3y ago
  Because a lot of the time, exactly how is very important. The devil is in the details.
  - pombo 3y ago
    I would say that it depends. Python and JS, two of the most used programming languages, abstract away memory management and threading details that most of the time you don't need, but you can "drop down" to write code that does pay attention to that within the language (building an object pool and reusing it, or memoization, etc) and if that's not enough, go to the "deeper" language like C/Rust to handle those cases with first-class primitives.
  - _ea1k 3y ago
    That's what all the C engineers told me about manual memory management. Of course, there are cases where they were right.
- pombo 3y ago
  I'm not sure I agree with the fact that an abstraction is always the answer, but your last sentence outlines precisely one of our motivations behind Marsha. You can specify an imperative set of steps in the description of a function/program, or you can (in the future) write a Python function that you use from Marsha
  - _ea1k 3y ago
    The first sentence was a little tongue in cheek, so I'm glad that you didn't 100% agree with it. :) But it was a all-too-common paradigm in my early years as a Java developer.
vector_spaces 3y ago
I don't think it is correct to call this a programming language. This program enforces some structure on your specifications for LLMs and provides some guardrails, which is absolutely a move in the right direction, but these are related more to formatting rather than specification, i.e. it provides syntax without transparent or unambiguous semantics. Ultimately this is more of a markup format than a programming language. Since: > The next section is the description of the function. Here you explain what the function should do. Being more explicit here will reduce variability in the generated output and improve reliability in behavior, but it's up to you just how explicit you will be and how much you leave to the LLM to figure out. There are reasons that natural language is problematic when specifying systems! I wish I could tape a sign that says this to the top of HN given the number of projects on the front page the past few months calling all sorts of things compilers that are just thin wrappers over the OpenAI API. These projects frame programming with natural language as though it is something desirable, like it is the holy grail of computer science that will open up engineering to the masses. But we use programming languages not to make programming difficult, but because unrestricted natural language is problematic. Systems are best specified with highly constrained languages that are (more) semantically unambiguous. Without sufficient constraints, there's a good chance that we don't even know what we ourselves are talking about when we specify systems -- how can we hope then that computers will? Even software engineers misinterpret and misunderstand requirements, requirements are poorly specified, or requirements are (apparently) well-understood but the space of possible inputs and use cases are not. This is why mathematicians use a (loosely) restricted jargon and notation, and even they run into difficulties All of that said: LLMs are surely a useful tool for software engineering and systems-building -- I personally use them most days to that end. But make no mistake that this is a markup language with some added guardrails to help users make better specifications for LLMs and verify the output. It is not a programming language, and programming with natural language is not generally something that is generally possible or even desirable
- Der_Einzige 3y ago
  If it's turing complete, it's a programming language. That's it. There's no other requirements.
sudosteph 3y ago
Is there a new trend to give AI/tech products common women's names, (Alexa, Macie, Ada, Clara, Julia, etc)? The only male named tech product I can think of really is Watson, and that's old hat by now. It honestly kind of weirds me out and I feel for people who have to share their names with a product. At least Siri and Cortana seem pretty unique in that respect. Fwiw, I'm sure plenty of products with traditionally male names exist, I just can't recall encountering many at work or casually.
- FireInsight 3y ago
  I think it's some weird manifestation of women being culturally more often in jobs as assistants to men and men in the leading roles. That and wanting the AI to seem approachable and human.
- swyx 3y ago
  there's also Jarvis and Alfred, on the male side, but yeah, it's a known cognitive bias to implicitly associate subservient things with female names. not much to do except be aware and try to correct for it. i think there's like 3 AI assistants named Samantha. obviously inspired by the movie, but does show a lack of imagination.
- gs17 3y ago
  ELIZA probably started it back in the 60s.
- ISV_Damocles 3y ago
  Well, a prior project we worked on was named Alan[1]. The choice was somewhat arbitrary: https://marsha.ai https://marsha.ai was available and we thought it was a fine name so here we are. [1]: https://alan-lang.org https://alan-lang.org
- bestcoder69 3y ago
  Hm, well, ChatGPT is a boy.
  - FireInsight 3y ago
chrisjj 3y ago
> Marsha uses this to provide more information to the LLM to generate the logic you want, but also uses it to generate a test suite to validate that what it has generated actually does what you want it to. What I can't find here is the component that reads "what you want it to" from the mind of the user.
andreygrehov 3y ago
Sharing a comment of mine (that got downvoted) from another, unrelated, thread. IMHO, it somewhat applies here as well: > Looking back, we can see how Machine Code, with its intricate and challenging nature, paved the way for more accessible options. Assembly language then emerged, providing a higher level of abstraction and reducing the complexities of directly working with machine instructions. And of course, C followed suit, offering even greater simplicity and ease of use compared to Assembly. > Imagine a future where programming languages, as we know them today, become akin to CPU instructions – a foundational and low-level primitive. LLMs will revolutionize the way we interact with code, providing a unified interface where the complexities of various languages are distilled into a common representation. The proliferation of individual programming languages will wane. Knowing Java or C++ will become a rare skill, akin to individuals specializing in low-level optimizations using Assembly language these days. > As time progresses, even the convenience of LLMs may pose challenges, given our inherent tendency towards laziness, so an additional layer of abstraction will be introduced, bridging the gap between LLMs and spoken languages. BCIs will revolutionize the act of coding itself so that individuals can seamlessly "code" by simply "thinking" about their desired actions.
- mathbalduino 3y ago
  Agree. It’s just that the majority of Hacker News users will resist such a major change Right now, it’s not that reliable, of course, but eventually it will be. As everything else, it will evolve Don’t think it will be production-ready in the next 10 years or so, but eventually it will happen… (I’ll save this reply to revisit in the future lol)
  - TheMode 3y ago
    > Right now, it’s not that reliable, of course, but eventually it will be. As everything else, it will evolve What make you think that natural languages will ever become reliable/stable? It hasn't been for millennia
    - flangola7 3y ago
      Then how have humans ever accomplished anything?
chadnorvell 3y ago
Check out maccarone (https://github.com/bsilverthorn/maccarone https://github.com/bsilverthorn/maccarone) for a different take on this concept.
roguecoder 3y ago
It is interesting to me how enthusiastic people are about natural language interfaces despite presumably having worked with requirements documents already (which are a programming language compiled by programmers). If we want our systems to behave gracefully and predictably, they aren't going to be built in natural language: they are going to be built in a system with locally-specific semantics. To build that with natural language, I would expect to see a conversational system that can request clarifications for ambiguity and negotiate local, specific definitions of the words it is using, similar to Domain Driven Design today. But for that to work, it will need semantic modeling not just the correlation in LLMs.
- voidhorse 3y ago
  This touches on how I've felt about this since the LLM craze started and people began exploring how to use it for programming. It's kind of amazing that we can define formal languages in which it's possible to specify provably correct behavior and now we want to....introduce randomness and greater possibility for unanticipated errors by generating code via LLMs? I feel like those pursuing LLMs basically lack respect for the inherent complexities of the many spheres of life. Not everything is reducible to linguistic pattern matching.
  - ISV_Damocles 3y ago
    I think you are painting with too broad of a brush. There are many domains that I would never use an LLM-based tool for; all tools can be used incorrectly, but that doesn't make the tool at fault. Software engineering is about trade-offs, for LLM-based code generation in general the trade-off is speeding up the writing of code at the expense of precision in what is generated. When you use something like Copilot it uses the comment or function signature to "guess" what you intend to write, and sometimes it right, sometimes it's not. Marsha is exploring that trade-off space. Copilot finishes in 5-20 seconds, usually, while Marsha's slower, sometimes as fast as 20 seconds, but usually a little over a minute. The syntax requires you to provide more information up front than just a comment or a function signature and also uses that up front information to generate a test suite to improve the reliability of what it outputs, which increases iterations with the LLM and therefore slows it down. Only when the code generated passes the test suite will it actually return an output to you, so the code it generated passes the cases that you were able to think of, which should make it much more precise than Copilot. That may still fail, but probably in ways your own code would have failed for cases you hadn't considered, so this particular trade-off feels closer to "free" versus writing it up by hand, in my opinion. But again, when to use the tool is a decision you must make. You can see from our own examples that we've only used it so far on toy problems or problems small enough that manual review is feasible. Since the test suite always passes at the end (or it simply fails to generate if not), that makes it better than many Eng I and some Eng II level engineers I have worked with in the past. ;)
ianbicking 3y ago
This reminds me a lot of AutoGPT and GPT-engineer and other "full automation" systems, only much more practical and reasonable. Like if I was going to try to make one of those self-programming systems work I'd be focused on getting it to outline and specify components, then divide that work on components, maybe doing revision passes, think about the specification order, etc. Many of those steps are things programmers naturally do and are supported by modules and functions and tests... So phrasing the specification language as a programming language instead of a single prompt is pretty reasonable. That said, I'm not sure if this needs to be its own language? Could it be in Python directly? (I'm reminded of my own magic Python module: https://ianbicking.org/blog/2023/01/infinite-ai-array.html https://ianbicking.org/blog/2023/01/infinite-ai-array.html – though honestly I haven't touched it since writing it... too much magic makes something not really useful.) I think it would help if the development process was maybe more incremental. I don't really use REPLs because I find it hard to construct and maintain the REPL environment. But maybe those compromises are more reasonable given the unique challenges of AI-written code. Like maybe the AI should be building mocks from the beginning, kind of gentle placeholders that get thrown away quickly, but since the AI wrote the original code no one really cares. I guess I'm proposing processes (architectures?) of construction that are specific to AI. Maybe not entirely novel, but the result of looking holistically at the development experience and picking out ideas that allow for incremental confirmation and refinement. AI is often better at making changes and refinements than creating something from whole cloth.
dd_xplore 3y ago
I never knew there was a Wikipedia package!
petabytes 3y ago
I just made something exactly like this a few weeks ago. My main goal was to see if I could self-host it. How long will it be before Marsha is self hosted?

140 points by ISV_Damocles 3y ago

ilaksh 3y ago

This is great. Have you thought about adding tree of thoughts?

pombo 3y ago
one of the contributors here. we have a statistical test script we can run on our branches to test the compile time, cost and reliability. we want to try tree of thought and also something like https://www.reddit.com/r/ChatGPT/comments/14d7pfz/become_god_like_prompt_engineer_with_this_one/ https://www.reddit.com/r/ChatGPT/comments/14d7pfz/become_god.... that said, we found that when we asked GPT to first explain why a test case is failing and then to correct that failure instead of just asking it to correct the failure, unexpectedly costs went up and reliability went down
- ISV_Damocles 3y ago
  Btw, here's the test job: https://github.com/alantech/marsha/blob/main/.github/workflows/time.yaml https://github.com/alantech/marsha/blob/main/.github/workflo... And the core script for the job: https://github.com/alantech/marsha/blob/main/marsha/.time.py https://github.com/alantech/marsha/blob/main/marsha/.time.py

franciscomello 3y ago

Sounds very interesting.

Winse 3y ago

I thought this would kind of just be a pile of garbage, but I have to admit I was drawn in. There are some interesting novel pieces in Marsha and I am somewhat impressed with this project.

ISV_Damocles 3y ago
yeah, the AI hype cycle can be a bit tiring, but I'm glad you took a second look. What parts did you find interesting, in particular?
weego 3y ago
The most interesting thing for me is them providing examples of the function made me realise this is the logical conclusion of TDD. Write tests to build the input -> output contracts, have AI build the logic that conforms to those contracts.

jstarfish 3y ago

Heh...and here I thought I was being clever using Inform 7 as a metalanguage. This is way more concise.

ISV_Damocles 3y ago
Well, to be fair, we do bulk up what you type quite a bit[1] to improve GPT's response. [1]: https://github.com/alantech/marsha/blob/main/marsha/parse.py#L114-L138 https://github.com/alantech/marsha/blob/main/marsha/parse.py...

cabalamat 3y ago

Why not just define the function headers in Python? It's less verbose.

ISV_Damocles 3y ago
Simply, Marsha is not Python. ;) More seriously, in Python, types are optional and the exact behavior is explicitly defined at runtime, but the author of the code already had a behavior in mind when they wrote the code. Getting the intended type information from the user to provide to the LLM improves the quality of the generated code, which is actually somewhat lossy. The "intended type information" isn't "explicit type information", though. We support any character besides these four as part of a type `(),:`. That includes spaces and periods, so you can describe the type as a short sentence if you aren't exactly sure what the structure ought to be.[1] And since in Marsha you don't actually write an imperative function body, argument names don't matter, only the argument order (as Marsha does not support named parameters, at least right now) so there's no argument naming in the declaration, which can make it more succinct than Python def lines in certain situations, though to be fair none of our examples do this. [1]: https://github.com/alantech/marsha/blob/main/examples/web/duckduckgo.mrsh#L1 https://github.com/alantech/marsha/blob/main/examples/web/du...

cabalamat 3y ago

How easy would it be to make it output programs in a functional language, such as Haskell? it would be nice if it could be made to produce functions that are guaranteed to be free from side effects. In the long term I can imagine that the output languages of code generators like this might be ones specially designed to work well with AI code generators.

ISV_Damocles 3y ago
So, targeting other languages is on our roadmap, and we have found higher reliability when we stick to a functional style with Marsha, so I think Haskell will be one of the "good ones" amongst target languages, eventually. But we are focusing on Python first because it's one of the most popular languages on the planet and many LLMs out there are able to generate high quality code for it, while less-used languages tend to produce worse results. Haskell may or may not need a manually-tuned LLM to generate solid results.
- cabalamat 3y ago
  > I think Haskell will be one of the "good ones" amongst target languages, eventually Another advantage of Haskell is that its side-effect-free nature makes it harder for a rogue AI to put in malicious code, thus helping to solve the alignment problem. It may be that languages and OSes will be specifically designed to put everything in a sandbox (or multiple layers of sandboxing), which will also be helpful when humans write code. > while less-used languages tend to produce worse results I figured that might be the case. > Haskell may or may not need a manually-tuned LLM to generate solid results. One possibility would be to write some Haskell programs, that illustrate paradigms in the language and train the LLM on them. Then get the LLM to solve similar problems, and automatically test how good its solutions are on grounds of solving the problem (passing test cases), performance (time and space used) and how easy it is for humans to understand the programs. Then use those solution to train the LLM to be better at Haskell, and give it another set of slightly harder problems to solve. In this way it would learn programming pretty much as humans do, by solving incrementally harder problems.

_ea1k 3y ago

This feels inevitable to me. All software engineering problems can be solved by the addition of another layer of abstraction. Why not abstract away the "how" layer and focus completely on the "what" layer? For a lot of data processing and integration problems, this would both eliminate a lot of work and increase reliability.

ModernMech 3y ago
Because a lot of the time, exactly how is very important. The devil is in the details.
- pombo 3y ago
  I would say that it depends. Python and JS, two of the most used programming languages, abstract away memory management and threading details that most of the time you don't need, but you can "drop down" to write code that does pay attention to that within the language (building an object pool and reusing it, or memoization, etc) and if that's not enough, go to the "deeper" language like C/Rust to handle those cases with first-class primitives.
- _ea1k 3y ago
  That's what all the C engineers told me about manual memory management. Of course, there are cases where they were right.
pombo 3y ago
I'm not sure I agree with the fact that an abstraction is always the answer, but your last sentence outlines precisely one of our motivations behind Marsha. You can specify an imperative set of steps in the description of a function/program, or you can (in the future) write a Python function that you use from Marsha
- _ea1k 3y ago
  The first sentence was a little tongue in cheek, so I'm glad that you didn't 100% agree with it. :) But it was a all-too-common paradigm in my early years as a Java developer.

vector_spaces 3y ago

I don't think it is correct to call this a programming language. This program enforces some structure on your specifications for LLMs and provides some guardrails, which is absolutely a move in the right direction, but these are related more to formatting rather than specification, i.e. it provides syntax without transparent or unambiguous semantics. Ultimately this is more of a markup format than a programming language. Since: > The next section is the description of the function. Here you explain what the function should do. Being more explicit here will reduce variability in the generated output and improve reliability in behavior, but it's up to you just how explicit you will be and how much you leave to the LLM to figure out. There are reasons that natural language is problematic when specifying systems! I wish I could tape a sign that says this to the top of HN given the number of projects on the front page the past few months calling all sorts of things compilers that are just thin wrappers over the OpenAI API. These projects frame programming with natural language as though it is something desirable, like it is the holy grail of computer science that will open up engineering to the masses. But we use programming languages not to make programming difficult, but because unrestricted natural language is problematic. Systems are best specified with highly constrained languages that are (more) semantically unambiguous. Without sufficient constraints, there's a good chance that we don't even know what we ourselves are talking about when we specify systems -- how can we hope then that computers will? Even software engineers misinterpret and misunderstand requirements, requirements are poorly specified, or requirements are (apparently) well-understood but the space of possible inputs and use cases are not. This is why mathematicians use a (loosely) restricted jargon and notation, and even they run into difficulties All of that said: LLMs are surely a useful tool for software engineering and systems-building -- I personally use them most days to that end. But make no mistake that this is a markup language with some added guardrails to help users make better specifications for LLMs and verify the output. It is not a programming language, and programming with natural language is not generally something that is generally possible or even desirable

Der_Einzige 3y ago
If it's turing complete, it's a programming language. That's it. There's no other requirements.

sudosteph 3y ago

Is there a new trend to give AI/tech products common women's names, (Alexa, Macie, Ada, Clara, Julia, etc)? The only male named tech product I can think of really is Watson, and that's old hat by now. It honestly kind of weirds me out and I feel for people who have to share their names with a product. At least Siri and Cortana seem pretty unique in that respect. Fwiw, I'm sure plenty of products with traditionally male names exist, I just can't recall encountering many at work or casually.

FireInsight 3y ago
I think it's some weird manifestation of women being culturally more often in jobs as assistants to men and men in the leading roles. That and wanting the AI to seem approachable and human.
swyx 3y ago
there's also Jarvis and Alfred, on the male side, but yeah, it's a known cognitive bias to implicitly associate subservient things with female names. not much to do except be aware and try to correct for it. i think there's like 3 AI assistants named Samantha. obviously inspired by the movie, but does show a lack of imagination.
gs17 3y ago
ELIZA probably started it back in the 60s.
ISV_Damocles 3y ago
Well, a prior project we worked on was named Alan[1]. The choice was somewhat arbitrary: https://marsha.ai https://marsha.ai was available and we thought it was a fine name so here we are. [1]: https://alan-lang.org https://alan-lang.org
bestcoder69 3y ago
Hm, well, ChatGPT is a boy.
- FireInsight 3y ago

chrisjj 3y ago

> Marsha uses this to provide more information to the LLM to generate the logic you want, but also uses it to generate a test suite to validate that what it has generated actually does what you want it to. What I can't find here is the component that reads "what you want it to" from the mind of the user.

andreygrehov 3y ago

Sharing a comment of mine (that got downvoted) from another, unrelated, thread. IMHO, it somewhat applies here as well: > Looking back, we can see how Machine Code, with its intricate and challenging nature, paved the way for more accessible options. Assembly language then emerged, providing a higher level of abstraction and reducing the complexities of directly working with machine instructions. And of course, C followed suit, offering even greater simplicity and ease of use compared to Assembly. > Imagine a future where programming languages, as we know them today, become akin to CPU instructions – a foundational and low-level primitive. LLMs will revolutionize the way we interact with code, providing a unified interface where the complexities of various languages are distilled into a common representation. The proliferation of individual programming languages will wane. Knowing Java or C++ will become a rare skill, akin to individuals specializing in low-level optimizations using Assembly language these days. > As time progresses, even the convenience of LLMs may pose challenges, given our inherent tendency towards laziness, so an additional layer of abstraction will be introduced, bridging the gap between LLMs and spoken languages. BCIs will revolutionize the act of coding itself so that individuals can seamlessly "code" by simply "thinking" about their desired actions.

mathbalduino 3y ago
Agree. It’s just that the majority of Hacker News users will resist such a major change Right now, it’s not that reliable, of course, but eventually it will be. As everything else, it will evolve Don’t think it will be production-ready in the next 10 years or so, but eventually it will happen… (I’ll save this reply to revisit in the future lol)
- TheMode 3y ago
  > Right now, it’s not that reliable, of course, but eventually it will be. As everything else, it will evolve What make you think that natural languages will ever become reliable/stable? It hasn't been for millennia
  - flangola7 3y ago
    Then how have humans ever accomplished anything?

chadnorvell 3y ago

Check out maccarone (https://github.com/bsilverthorn/maccarone https://github.com/bsilverthorn/maccarone) for a different take on this concept.

roguecoder 3y ago

It is interesting to me how enthusiastic people are about natural language interfaces despite presumably having worked with requirements documents already (which are a programming language compiled by programmers). If we want our systems to behave gracefully and predictably, they aren't going to be built in natural language: they are going to be built in a system with locally-specific semantics. To build that with natural language, I would expect to see a conversational system that can request clarifications for ambiguity and negotiate local, specific definitions of the words it is using, similar to Domain Driven Design today. But for that to work, it will need semantic modeling not just the correlation in LLMs.

voidhorse 3y ago
This touches on how I've felt about this since the LLM craze started and people began exploring how to use it for programming. It's kind of amazing that we can define formal languages in which it's possible to specify provably correct behavior and now we want to....introduce randomness and greater possibility for unanticipated errors by generating code via LLMs? I feel like those pursuing LLMs basically lack respect for the inherent complexities of the many spheres of life. Not everything is reducible to linguistic pattern matching.
- ISV_Damocles 3y ago
  I think you are painting with too broad of a brush. There are many domains that I would never use an LLM-based tool for; all tools can be used incorrectly, but that doesn't make the tool at fault. Software engineering is about trade-offs, for LLM-based code generation in general the trade-off is speeding up the writing of code at the expense of precision in what is generated. When you use something like Copilot it uses the comment or function signature to "guess" what you intend to write, and sometimes it right, sometimes it's not. Marsha is exploring that trade-off space. Copilot finishes in 5-20 seconds, usually, while Marsha's slower, sometimes as fast as 20 seconds, but usually a little over a minute. The syntax requires you to provide more information up front than just a comment or a function signature and also uses that up front information to generate a test suite to improve the reliability of what it outputs, which increases iterations with the LLM and therefore slows it down. Only when the code generated passes the test suite will it actually return an output to you, so the code it generated passes the cases that you were able to think of, which should make it much more precise than Copilot. That may still fail, but probably in ways your own code would have failed for cases you hadn't considered, so this particular trade-off feels closer to "free" versus writing it up by hand, in my opinion. But again, when to use the tool is a decision you must make. You can see from our own examples that we've only used it so far on toy problems or problems small enough that manual review is feasible. Since the test suite always passes at the end (or it simply fails to generate if not), that makes it better than many Eng I and some Eng II level engineers I have worked with in the past. ;)

ianbicking 3y ago

This reminds me a lot of AutoGPT and GPT-engineer and other "full automation" systems, only much more practical and reasonable. Like if I was going to try to make one of those self-programming systems work I'd be focused on getting it to outline and specify components, then divide that work on components, maybe doing revision passes, think about the specification order, etc. Many of those steps are things programmers naturally do and are supported by modules and functions and tests... So phrasing the specification language as a programming language instead of a single prompt is pretty reasonable. That said, I'm not sure if this needs to be its own language? Could it be in Python directly? (I'm reminded of my own magic Python module: https://ianbicking.org/blog/2023/01/infinite-ai-array.html https://ianbicking.org/blog/2023/01/infinite-ai-array.html – though honestly I haven't touched it since writing it... too much magic makes something not really useful.) I think it would help if the development process was maybe more incremental. I don't really use REPLs because I find it hard to construct and maintain the REPL environment. But maybe those compromises are more reasonable given the unique challenges of AI-written code. Like maybe the AI should be building mocks from the beginning, kind of gentle placeholders that get thrown away quickly, but since the AI wrote the original code no one really cares. I guess I'm proposing processes (architectures?) of construction that are specific to AI. Maybe not entirely novel, but the result of looking holistically at the development experience and picking out ideas that allow for incremental confirmation and refinement. AI is often better at making changes and refinements than creating something from whole cloth.

dd_xplore 3y ago

I never knew there was a Wikipedia package!

petabytes 3y ago

I just made something exactly like this a few weeks ago. My main goal was to see if I could self-host it. How long will it be before Marsha is self hosted?