3 ms·
Do you have any research, empirical data, or other hard evidence that shows that a model (especially a non-SOTA model) makes engineers even "a little faster"? I
by angoragoats 4mo ago
Do you have any research, empirical data, or other hard evidence that shows that a model (especially a non-SOTA model) makes engineers even "a little faster"? I'm aware of anecdotes, but nothing more.
- missedthecue 4mo agoThe cleanest positive study is Cui et al., because it used randomized rollout in real companies. Developers at Microsoft, Accenture, and a Fortune 100 electronics manufacturer were randomly given access to an AI coding assistant. Pooled across 4,867 developers, the authors estimate a 26.08% increase in completed tasks among users. They also report increases in commits and builds. While that research was published in 2026, it analyzed the 2023 period. Very much NOT sota models! https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535 https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535
- angoragoats 4mo agoThank you!