5 ms·
"Airbnb recently completed our first large-scale, LLM-driven code migration, updating nearly 3.5K React component test files from Enzyme to use React Testing Li
by mech422 1y ago
"Airbnb recently completed our first large-scale, LLM-driven code migration, updating nearly 3.5K React component test files from Enzyme to use React Testing Library (RTL) instead. We’d originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks."
Color me unimpressed - it converted some test files. It didn't design any architecture, create any databases, handle any security concerns or any of the other things programmers have to do/worry about on a daily basis. It basically did source to source translation, which has been around for 30+ years.
- shermantanktop 1y agoIf you told me five years ago that such a conversion had been done in six weeks, I would not have believed it. Even though some level of source-to-source existed. And I would definitely expect that such a conversion would have resulted in hideous, non-idiomatic code in the target language.
- riku_iki 1y ago> And I would definitely expect that such a conversion would have resulted in hideous, non-idiomatic code in the target language. and we don't know what is the quality of end code, it is possible that tech debt created by migration is well higher than 1.5 eng/y.
- bobxmax 1y agoDo you really think Airbnb software engineers would accept the final code if it was low quality? Seriously, the wishful thinking around this stuff is embarassing.
- riku_iki 1y agoThere are plenty of tech bros in corps who push some hyped stuff, add to resume and move to next gig. Not sure why do you think airbnb is different.
- shermantanktop 1y agoHave you tried it? I’ve done something similar to this type of work with an llm. It produces code that is often too idiomatic, in that it introduces conventional approaches that are overkill for the task at hand. But this is almost an ideal scenario, because if the previous tests run clean, the newly converted tests can only fail by being wrong. They can also silently reduce the tested scenarios, but that’s what code reviews are for.
- riku_iki 1y agoI use LLM for coding every day. LLMs can totally make something super dumb: oh this test is broken, gatcha, lets replace it with something which is not broken (but very different). > They can also silently reduce the tested scenarios, but that’s what code reviews are for. code review of complicated scenarios is about as resource demanding as actually writing code, so shrinking time spent from 1.5y to 6 weeks totally can produce lower quality.
- bobxmax 1y agoIt's easy to tell yourself you're not losing by constantly pushing the goal posts.
- mech422 1y agogreat comeback - "Dude - Trust me" :-P got anything more compelling then your opinion ? Have you ever done source to source translation work (I have ...) Or is this just so exciting to you cuz you've never seen it before ?
- bobxmax 1y agoI didn't say anything remotely close to "Dude Trust Me" - perhaps you need to try a bit of "vibe learning how to read english"
- mech422 1y agoexactly, you didn't provide any evidence or even decent arguements to support your position. I provided 3 distinct areas (of many) that it didn't touch that programmers have to deal with everyday. I also pointed out that the capabilities to do the sorts of transformations it did have existed for decades. But given your comment history, backing up your arguments doesn't seem to be your strong suit...