3 ms·

Anthropic runs large-scale code migrations with Claude Code

36 points by vinhnx 3mo ago

grandimam 3mo ago
[flagged]
SpicyLemonZest 3mo ago
> Jarred Sumner, co-founder of Bun and Member of Technical Staff at Anthropic, used Claude Code to migrate Bun from Zig to Rust. A million lines of code were produced in less than two weeks, with 100% of Bun's existing test suite passing in CI before merge. Nineteen regressions surfaced after merge and have all been fixed. I couldn't come up with a clearer example of how this kind of stuff fails to apply to my work. If I shipped nineteen regressions in 2 weeks, my company would be desperately apologizing to our customers, not writing a blog post about what a great job I did.
- JackSlateur 3mo ago
  I guess even anthropic fails to not get slop from ai ..
- Salgat 3mo ago
  Regressions is a meaningless metric to me. Every app has bugs, and most are either rarely encountered or aren't breaking the app. All I care about is interruption of service.
- IshKebab 3mo ago
  Umm yeah because you didn't rewrite a million line codebase in Rust in two weeks. Obviously "regressions" is not a great measure (how serious were they?) but 19 seems pretty decent to me.
- jryle70 3mo ago
  They didn't ship 19 regressions over 2 weeks. "Merged" doesn't mean "shipped". Their customers have probably seen none of those because they've all been fixed. Such a terribly bad faith take. I feel upset reading this more than I should have.
  - SpicyLemonZest 3mo ago
    On what basis do you say their customers probably didn't see them? I'm not intimately familiar with Bun, but I work with people who try to do software development in this way, and they routinely ship regressions to customers. They believe that this is because we're not AI-enabled enough, and if only we had the right loops they'd stop shipping bugs; I think that it's not about the loops and the bugs reflect their lack of understanding about what they're trying to do.
    - throw1234567891 3mo ago
      They told you, can’t you read? All 19 regressions were fixed before release.
      - SpicyLemonZest 3mo ago
        The source article says no such thing and at least one of them (https://github.com/oven-sh/bun/issues/31188 https://github.com/oven-sh/bun/issues/31188) seems to have been publicly reported.
        scrollaway 3mo ago
        It was fixed on May 21st, which is 6 days after it was merged to main, and is 6.5 weeks before the release.
        nickv 3mo ago
        Dude, just give up and admit you misread the point in the most negative bad faith way possible.
        throw1234567891 3mo ago
        Because people cannot try open source software by building it themselves before release, right? No?
        deleted 3mo ago
- scrollaway 3mo ago
  Migrating a million lines of code is how many human-engineer-years worth of work? Some quick fermi estimates place it at between 15-100 years depending on how much automated tooling is used (without AI). It really blows my mind how some AI naysayers like you fail the absolute basic common sense test of comparing the comparable. If you, without AI, merged nineteen regressions in 2 weeks, how much work has actually been done? How many features did you rewrite? How many refactors did you make and how difficult were they? Insane...
  - SpicyLemonZest 3mo ago
    It's precisely because I'm not an AI naysayer that I reject the premise of the comparison. My job is not to produce as much code as I possibly can; my job is to produce features that provide value to customers in a high quality and regression-free manner. To accomplish this I use safer AI-enabled development strategies, which produce fewer lines of code and involve more manual review but also do not result in 1 regression per calendar day.
    - scrollaway 3mo ago
      Again, you are comparing things which are not comparable. "Working on a feature" and "Working on a language-to-language port of 1M LOC" are NOT the same thing. You're also still bringing up "regression per day", while ignoring the fact that without AI, this would have taken easily ~4000 days. Which is 1 regression per 0.003 calendar day. Not the same. And you still aren't addressing sibling comments reminding you that these aren't shipped migrations. I'll second the comment asking you whether your code is always perfect first-shot, because if we're going to count those bugs, we might be well above your 1/day number.
      - well_ackshually 3mo ago
        No, without AI, it wouldn't have been done, which would have led to exactly zero bugs added by a needless rewrite.
  - rurban 3mo ago
- entrope 3mo ago
  The regressions got fixed before a release. Do you count your internal integration branch as "shipped"? 0.02 defects per KLOC is a low density, even considering that it was "just" a port to a new language and before giving credit for the existing bugs it identified and fixed. That defect density will probably only generalize to other projects that have similarly extensive regression test suites, but it's a proof of what LLMs can do.
  - SpicyLemonZest 3mo ago
    The source article says that "Large code migrations are a particularly effective use case for these advanced models", and the specific advantages suggest to me that the results won't generalize at all. I can't think of anything other than a large code migration that would allow for thousands of independent workstreams against a sufficiently detailed spec to stop agents from going completely off track.
watwut 3mo ago
So, the whole brouhaha was so that Anthropic can write this blog post?
- bwfan123 3mo ago
  > So, the whole brouhaha was so that Anthropic can write this blog post? Remember mythos ? I was half-expecting them to scare us with a new danger: How AI can covert code into other languages, and confuse the hell out of all of us.
- jryle70 3mo ago
  So much complaints about wasting of energy for AI, yet how much energy does such a meaningless comment waste?
  - watwut 3mo ago
    Not much, this was both accurate comment and easy to make comment. And it is good for mental health - it allows us to express the distaste for Anthropic in a healhy way.
  - nozzlegear 3mo ago
    Goomba fallacy
- whateveracct 3mo ago
  yes the whole thing was partially a big marketing stunt
- nozzlegear 3mo ago
  > So, the whole brouhaha was so that Anthropic can write this blog post? This was suspected from the start. The PR was always going to be merged, and Anthropic was going to have their success story. Any lip service paid to "well maybe we'll merge it and maybe we won't, I dunno yet" was just that: lip service.
londons_explore 3mo ago
So I can expect an entirely-rust Linux kernel to be available next week then?
- yehosef 3mo ago
  if Linus has unrestricted access to Mythos.
  - taskforcegemini 3mo ago
    or anthropic forks linux. one day ai may fork us all

36 points by vinhnx 3mo ago

[flagged]

> Jarred Sumner, co-founder of Bun and Member of Technical Staff at Anthropic, used Claude Code to migrate Bun from Zig to Rust. A million lines of code were produced in less than two weeks, with 100% of Bun's existing test suite passing in CI before merge. Nineteen regressions surfaced after merge and have all been fixed. I couldn't come up with a clearer example of how this kind of stuff fails to apply to my work. If I shipped nineteen regressions in 2 weeks, my company would be desperately apologizing to our customers, not writing a blog post about what a great job I did.

JackSlateur 3mo ago
I guess even anthropic fails to not get slop from ai ..
Salgat 3mo ago
Regressions is a meaningless metric to me. Every app has bugs, and most are either rarely encountered or aren't breaking the app. All I care about is interruption of service.
IshKebab 3mo ago
Umm yeah because you didn't rewrite a million line codebase in Rust in two weeks. Obviously "regressions" is not a great measure (how serious were they?) but 19 seems pretty decent to me.
jryle70 3mo ago
They didn't ship 19 regressions over 2 weeks. "Merged" doesn't mean "shipped". Their customers have probably seen none of those because they've all been fixed. Such a terribly bad faith take. I feel upset reading this more than I should have.
- SpicyLemonZest 3mo ago
  On what basis do you say their customers probably didn't see them? I'm not intimately familiar with Bun, but I work with people who try to do software development in this way, and they routinely ship regressions to customers. They believe that this is because we're not AI-enabled enough, and if only we had the right loops they'd stop shipping bugs; I think that it's not about the loops and the bugs reflect their lack of understanding about what they're trying to do.
  - throw1234567891 3mo ago
    They told you, can’t you read? All 19 regressions were fixed before release.
    - SpicyLemonZest 3mo ago
      The source article says no such thing and at least one of them (https://github.com/oven-sh/bun/issues/31188 https://github.com/oven-sh/bun/issues/31188) seems to have been publicly reported.
      - scrollaway 3mo ago
        It was fixed on May 21st, which is 6 days after it was merged to main, and is 6.5 weeks before the release.
      - nickv 3mo ago
        Dude, just give up and admit you misread the point in the most negative bad faith way possible.
      - throw1234567891 3mo ago
        Because people cannot try open source software by building it themselves before release, right? No?
        deleted 3mo ago
scrollaway 3mo ago
Migrating a million lines of code is how many human-engineer-years worth of work? Some quick fermi estimates place it at between 15-100 years depending on how much automated tooling is used (without AI). It really blows my mind how some AI naysayers like you fail the absolute basic common sense test of comparing the comparable. If you, without AI, merged nineteen regressions in 2 weeks, how much work has actually been done? How many features did you rewrite? How many refactors did you make and how difficult were they? Insane...
- SpicyLemonZest 3mo ago
  It's precisely because I'm not an AI naysayer that I reject the premise of the comparison. My job is not to produce as much code as I possibly can; my job is to produce features that provide value to customers in a high quality and regression-free manner. To accomplish this I use safer AI-enabled development strategies, which produce fewer lines of code and involve more manual review but also do not result in 1 regression per calendar day.
  - scrollaway 3mo ago
    Again, you are comparing things which are not comparable. "Working on a feature" and "Working on a language-to-language port of 1M LOC" are NOT the same thing. You're also still bringing up "regression per day", while ignoring the fact that without AI, this would have taken easily ~4000 days. Which is 1 regression per 0.003 calendar day. Not the same. And you still aren't addressing sibling comments reminding you that these aren't shipped migrations. I'll second the comment asking you whether your code is always perfect first-shot, because if we're going to count those bugs, we might be well above your 1/day number.
    - well_ackshually 3mo ago
      No, without AI, it wouldn't have been done, which would have led to exactly zero bugs added by a needless rewrite.
- rurban 3mo ago
entrope 3mo ago
The regressions got fixed before a release. Do you count your internal integration branch as "shipped"? 0.02 defects per KLOC is a low density, even considering that it was "just" a port to a new language and before giving credit for the existing bugs it identified and fixed. That defect density will probably only generalize to other projects that have similarly extensive regression test suites, but it's a proof of what LLMs can do.
- SpicyLemonZest 3mo ago
  The source article says that "Large code migrations are a particularly effective use case for these advanced models", and the specific advantages suggest to me that the results won't generalize at all. I can't think of anything other than a large code migration that would allow for thousands of independent workstreams against a sufficiently detailed spec to stop agents from going completely off track.

watwut 3mo ago

So, the whole brouhaha was so that Anthropic can write this blog post?

bwfan123 3mo ago
> So, the whole brouhaha was so that Anthropic can write this blog post? Remember mythos ? I was half-expecting them to scare us with a new danger: How AI can covert code into other languages, and confuse the hell out of all of us.
jryle70 3mo ago
So much complaints about wasting of energy for AI, yet how much energy does such a meaningless comment waste?
- watwut 3mo ago
  Not much, this was both accurate comment and easy to make comment. And it is good for mental health - it allows us to express the distaste for Anthropic in a healhy way.
- nozzlegear 3mo ago
  Goomba fallacy
whateveracct 3mo ago
yes the whole thing was partially a big marketing stunt
nozzlegear 3mo ago
> So, the whole brouhaha was so that Anthropic can write this blog post? This was suspected from the start. The PR was always going to be merged, and Anthropic was going to have their success story. Any lip service paid to "well maybe we'll merge it and maybe we won't, I dunno yet" was just that: lip service.

londons_explore 3mo ago

So I can expect an entirely-rust Linux kernel to be available next week then?

yehosef 3mo ago
if Linus has unrestricted access to Mythos.
- taskforcegemini 3mo ago
  or anthropic forks linux. one day ai may fork us all