4 ms·

I think a lot of people taking this at face value , a lot of this was possible because of the beyond standard extensive and comprehensive test suit previously b

by ksec 6mo ago

I think a lot of people taking this at face value , a lot of this was possible because of the beyond standard extensive and comprehensive test suit previously built.

scuff3d 6mo ago
Look what it can do in 6 days! Ignore the hundreds of thousands of hours put into the original architecture and test suite that made it possible in the first place.
- cmrdporcupine 6mo ago
  Exactly this. I am not sure why people sound so astounded, to be honest. This has been my frank experience of the agentic tools both Codex and Claude since about December. When given the right constraints this kind of thing is entirely conceivable. However the important question not being answered here is: does anybody working on it have a full understanding of what has been built? My experience having constructed similar types of projects using these tools is yes, you could do this in a week or two but now you'll have a month or two of digging through what it made, understanding what was built, and undoing critical yolo leaps of faith it made that you didn't want.
  - scuff3d 6mo ago
    Not to mention to even attempt something like this from scratch would take hundreds of hours if spec work. I see it all day everyday in the aerospace sector. Software engineers have absolutely no idea what deriving a design document and all its associated artifacts actually looks like, and they're in for a rude surprise if the industry really does shift hard that direction
- zaptheimpaler 6mo ago
  This is such a bad faith argument. How long would it take a dev or a team of devs to do this with the same architecture and test suite? A hell of a lot longer than 6 days..
  - oytis 6mo ago
    But what is the purpose? When you rewrite a project in another language, it's for engineers to be able to maintain and further develop the project better on some metrics due to advantages of the language. It doesn't hold when LLM does the rewrite, since there is no one who understands the code after that. It's a good demonstration of capabilities, sure, but the result itself makes no sense. We'll have to figure out where these capabilities can bring real advantage
    - gamegod 6mo ago
      This is such an insightful comment. It also underscores why these AI companies' marketing efforts are promoting rewrites.
      - stingraycharles 6mo ago
        I agree that the comment is insightful, but I don’t think AI companies are particularly promoting rewrites, other than that it’s a task LLMs are good at as “the code is the spec”. The industry as a whole still is realizing that any LLM usage that actually writes all the code for you is causing cognitive debt, and we’re even slowly losing our skills of the art. I’m trying my best to navigate this myself, but no matter what we do, using LLMs is both a blessing and a curse.
    - cdelsolar 6mo ago
      why do you think no one understands the code after the LLM rewrites it?
      - oytis 6mo ago
        Becase no one has written it. You can't ask the guy who has written it, not because this guy has left, but because he does not exist. Also, it often reads weirdly.
        cdelsolar 6mo ago
Jcampuzano2 6mo ago
It's still an impressive achievement that would have taken even the most competent engineers an exponentially longer time to accomplish. I just hope it's noted when this is eventually marketed how much human effort went into designing and curating the test suite that even enabled this speed in the first place. A test suite sort of functions exactly like the ideal scenario for current gen llms. A comprehensive enough test suite essentially forms the spec for agents to implement however they see fit - in this case rust. You could probably throw away the entire actual source code in certain cases and reimplement the whole thing from scratch just giving an agent access to the tests when it's as well crafted as a project like bun.
leecommamichael 6mo ago
If this is a "beyond standard" test suite, (so much so that it _uniquely_ makes this work possible compared to other projects,) then how is Bun also uniquely unstable compared to other Zig programs (and so deserving of rewrite?) If the blame lies partially with the test suite, what does this imply (if anything) about the Rust port?
- stingraycharles 6mo ago
  Because tests validate behavior, not undefined behavior. The thesis is that Rust makes undefined behavior less likely.

by ksec 6mo ago

I think a lot of people taking this at face value , a lot of this was possible because of the beyond standard extensive and comprehensive test suit previously built.

scuff3d 6mo ago

Look what it can do in 6 days! Ignore the hundreds of thousands of hours put into the original architecture and test suite that made it possible in the first place.

cmrdporcupine 6mo ago
Exactly this. I am not sure why people sound so astounded, to be honest. This has been my frank experience of the agentic tools both Codex and Claude since about December. When given the right constraints this kind of thing is entirely conceivable. However the important question not being answered here is: does anybody working on it have a full understanding of what has been built? My experience having constructed similar types of projects using these tools is yes, you could do this in a week or two but now you'll have a month or two of digging through what it made, understanding what was built, and undoing critical yolo leaps of faith it made that you didn't want.
- scuff3d 6mo ago
  Not to mention to even attempt something like this from scratch would take hundreds of hours if spec work. I see it all day everyday in the aerospace sector. Software engineers have absolutely no idea what deriving a design document and all its associated artifacts actually looks like, and they're in for a rude surprise if the industry really does shift hard that direction
zaptheimpaler 6mo ago
This is such a bad faith argument. How long would it take a dev or a team of devs to do this with the same architecture and test suite? A hell of a lot longer than 6 days..
- oytis 6mo ago
  But what is the purpose? When you rewrite a project in another language, it's for engineers to be able to maintain and further develop the project better on some metrics due to advantages of the language. It doesn't hold when LLM does the rewrite, since there is no one who understands the code after that. It's a good demonstration of capabilities, sure, but the result itself makes no sense. We'll have to figure out where these capabilities can bring real advantage
  - gamegod 6mo ago
    This is such an insightful comment. It also underscores why these AI companies' marketing efforts are promoting rewrites.
    - stingraycharles 6mo ago
      I agree that the comment is insightful, but I don’t think AI companies are particularly promoting rewrites, other than that it’s a task LLMs are good at as “the code is the spec”. The industry as a whole still is realizing that any LLM usage that actually writes all the code for you is causing cognitive debt, and we’re even slowly losing our skills of the art. I’m trying my best to navigate this myself, but no matter what we do, using LLMs is both a blessing and a curse.
  - cdelsolar 6mo ago
    why do you think no one understands the code after the LLM rewrites it?
    - oytis 6mo ago
      Becase no one has written it. You can't ask the guy who has written it, not because this guy has left, but because he does not exist. Also, it often reads weirdly.
      - cdelsolar 6mo ago

Jcampuzano2 6mo ago

It's still an impressive achievement that would have taken even the most competent engineers an exponentially longer time to accomplish. I just hope it's noted when this is eventually marketed how much human effort went into designing and curating the test suite that even enabled this speed in the first place. A test suite sort of functions exactly like the ideal scenario for current gen llms. A comprehensive enough test suite essentially forms the spec for agents to implement however they see fit - in this case rust. You could probably throw away the entire actual source code in certain cases and reimplement the whole thing from scratch just giving an agent access to the tests when it's as well crafted as a project like bun.

leecommamichael 6mo ago

If this is a "beyond standard" test suite, (so much so that it _uniquely_ makes this work possible compared to other projects,) then how is Bun also uniquely unstable compared to other Zig programs (and so deserving of rewrite?) If the blame lies partially with the test suite, what does this imply (if anything) about the Rust port?

stingraycharles 6mo ago
Because tests validate behavior, not undefined behavior. The thesis is that Rust makes undefined behavior less likely.