4 ms·

I don't think AI lives up to the current hype but this article is garbage. They're obviously talking about the METR paper, but the main takeaway according to t

by qsort 1y ago

I don't think AI lives up to the current hype but this article is garbage. They're obviously talking about the METR paper, but the main takeaway according to the authors themselves was that self-reporting productivity increases is unreliable, not that you should cancel your subscription. Nothing in that paper said that AI can't speed up software engineering. Why are we responding to hype with nonsense?

didibus 1y ago
> Nothing in that paper said that AI can't speed up software engineering I mean, the paper did provide tangible data that at least in their experiments, AI slowed down software engineering. What they said is that it's not a proof that there's isn't a scenario or a mechanism where AI could result in speeding up software engineering. For that more research would be needed in measuring productivity of AI in more varied contexts. For me at least, their experiment seem to describe the average developer's use of AI. So it's probably telling you that currently on average AI might be slowing things down. Now the question is, can we find good data of outliers, and is it a simple matter of figuring out how to use it effectively, so we can upskill people and get the average to now be faster. Or will the outlier be conditioned on like, only for newbies, only for prototypes, only for the first X weeks on a greenfield code base, etc. Edit: That said, the most fascinating data point of that study is how software engineers are not able to determine if AI makes them faster or slower, because they all thought they were 20% faster but were 19% slower in reality. So now you have to become really skeptical of anyone who claims they found a methodology or a workflow where their use of AI makes them faster. We need better measurement than just "I feel faster".

4 ms·

I don't think AI lives up to the current hype but this article is garbage. They're obviously talking about the METR paper, but the main takeaway according to t

by qsort 1y ago

didibus 1y ago
> Nothing in that paper said that AI can't speed up software engineering I mean, the paper did provide tangible data that at least in their experiments, AI slowed down software engineering. What they said is that it's not a proof that there's isn't a scenario or a mechanism where AI could result in speeding up software engineering. For that more research would be needed in measuring productivity of AI in more varied contexts. For me at least, their experiment seem to describe the average developer's use of AI. So it's probably telling you that currently on average AI might be slowing things down. Now the question is, can we find good data of outliers, and is it a simple matter of figuring out how to use it effectively, so we can upskill people and get the average to now be faster. Or will the outlier be conditioned on like, only for newbies, only for prototypes, only for the first X weeks on a greenfield code base, etc. Edit: That said, the most fascinating data point of that study is how software engineers are not able to determine if AI makes them faster or slower, because they all thought they were 20% faster but were 19% slower in reality. So now you have to become really skeptical of anyone who claims they found a methodology or a workflow where their use of AI makes them faster. We need better measurement than just "I feel faster".