4 ms·

I'm curious about that too. I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wond

by jdw64 2mo ago

I'm curious about that too. I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wonder what kind of structure AI-only code would have.

ivan888 2mo ago
Is it like self driving cars, where 100% non-human code would be arguably safer? But then it always comes down to having some human who is accountable for verifying the behavior. I think ultimately we just need a better set of languages and frameworks to minimally encode the types of applications we want to build today, so that human comprehension and verification of the source code becomes easier.
- LetsGetTechnicl 2mo ago
  How would 100% non-human code possibly be safer?
  - micromacrofoot 2mo ago
    well if driving is supposedly safer when automated, why not code
    - ModernMech 2mo ago
      It’s supposedly safer than the average driver, not safer than professional drivers.
      - micromacrofoot 2mo ago
        the average american spends 60 minutes a day driving, that's thousands of hours by the time you reach 25... at what point is that not a professional level of experience?
    - dd8601fn 2mo ago
      The qualities of computation that make autopilots safer than human drivers aren’t the same qualities that produce better designed software products.
      - micromacrofoot 2mo ago
        to err is human
    - LetsGetTechnicl 2mo ago
      Is it actually safer? If you can't understand the code you can't verify it's safety, and AI code often has bugs in it. You can't trust implicitly.
      - skeledrew 2mo ago
        You don't need to understand code to verify its safety if you have a good QA framework.
      - micromacrofoot 2mo ago
        Same applies to driving cars
- jdw64 2mo ago
  I've noticed that when only AI writes the code, there are cases where it ends up being unusable by humans. For small code snippets or function-level units, AI is often much better than me. But when I look at the whole picture, AI tends to perform poorly.
- roncesvalles 2mo ago
  Category error. Self-driving cars are still meticulously coded by humans.
  - fragmede 2mo ago
    They use trained models in the system though, so I don't know if you can really claim that.
onlyrealcuzzo 2mo ago
Also building a language with AI. It's decent at design and implementation, but strangely terrible at language consistency - and expectedly architecture. Steering it on language consistency is easy. Steering it on architecture has required me to learn a lot more about compilers than I wanted to do for this project (since that's a lot more than I wanted to take on for something that was supposed to be a "fun" "side" project).
- jdw64 2mo ago
  I have a similar problem. I'm not sure about architecture. When I first implemented the slot system in my language, I designed it around that implementation, and in the end, the architecture suffered. There were quite a few cases where consistency was simply unavoidable. It's not easy to create a completely new architecture with something that doesn't exist in existing languages. Consistency doesn't come together as easily as you'd think. In that sense, I really admire the people who first introduced something new. When I started implementing it, I focused on one killer feature and just went for it—with the feeling that if this succeeds, my language will succeed. But once I started building around it, it turned out to be very different from other compiler architectures. With CRUD or other well-established domains, it's easier to maintain consistency. But achieving internal consistency in a program seems to be a different kind of challenge.
lonelyasacloud 2mo ago
> I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. In the linked paper the researchers state "AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. " And a bit further on "The AI agent is given execute-only access to the original program and a set of visible test cases, but does not have access to the original source code. " At a guess, their agents are not going off into the weeds (like mine do occasionally) is because they have both access to a reference runtime implementation to use as a definitive end to end test to guide development. And also a relatively large token budget to run enough characterisation tests to allow them to home in on solutions.

by jdw64 2mo ago

ivan888 2mo ago

Is it like self driving cars, where 100% non-human code would be arguably safer? But then it always comes down to having some human who is accountable for verifying the behavior. I think ultimately we just need a better set of languages and frameworks to minimally encode the types of applications we want to build today, so that human comprehension and verification of the source code becomes easier.

LetsGetTechnicl 2mo ago
How would 100% non-human code possibly be safer?
- micromacrofoot 2mo ago
  well if driving is supposedly safer when automated, why not code
  - ModernMech 2mo ago
    It’s supposedly safer than the average driver, not safer than professional drivers.
    - micromacrofoot 2mo ago
      the average american spends 60 minutes a day driving, that's thousands of hours by the time you reach 25... at what point is that not a professional level of experience?
  - dd8601fn 2mo ago
    The qualities of computation that make autopilots safer than human drivers aren’t the same qualities that produce better designed software products.
    - micromacrofoot 2mo ago
      to err is human
  - LetsGetTechnicl 2mo ago
    Is it actually safer? If you can't understand the code you can't verify it's safety, and AI code often has bugs in it. You can't trust implicitly.
    - skeledrew 2mo ago
      You don't need to understand code to verify its safety if you have a good QA framework.
    - micromacrofoot 2mo ago
      Same applies to driving cars
jdw64 2mo ago
I've noticed that when only AI writes the code, there are cases where it ends up being unusable by humans. For small code snippets or function-level units, AI is often much better than me. But when I look at the whole picture, AI tends to perform poorly.
roncesvalles 2mo ago
Category error. Self-driving cars are still meticulously coded by humans.
- fragmede 2mo ago
  They use trained models in the system though, so I don't know if you can really claim that.

onlyrealcuzzo 2mo ago

Also building a language with AI. It's decent at design and implementation, but strangely terrible at language consistency - and expectedly architecture. Steering it on language consistency is easy. Steering it on architecture has required me to learn a lot more about compilers than I wanted to do for this project (since that's a lot more than I wanted to take on for something that was supposed to be a "fun" "side" project).

jdw64 2mo ago
I have a similar problem. I'm not sure about architecture. When I first implemented the slot system in my language, I designed it around that implementation, and in the end, the architecture suffered. There were quite a few cases where consistency was simply unavoidable. It's not easy to create a completely new architecture with something that doesn't exist in existing languages. Consistency doesn't come together as easily as you'd think. In that sense, I really admire the people who first introduced something new. When I started implementing it, I focused on one killer feature and just went for it—with the feeling that if this succeeds, my language will succeed. But once I started building around it, it turned out to be very different from other compiler architectures. With CRUD or other well-established domains, it's easier to maintain consistency. But achieving internal consistency in a program seems to be a different kind of challenge.

lonelyasacloud 2mo ago

> I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. In the linked paper the researchers state "AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. " And a bit further on "The AI agent is given execute-only access to the original program and a set of visible test cases, but does not have access to the original source code. " At a guess, their agents are not going off into the weeds (like mine do occasionally) is because they have both access to a reference runtime implementation to use as a definitive end to end test to guide development. And also a relatively large token budget to run enough characterisation tests to allow them to home in on solutions.