3 ms·
I think the 8% number better explains why users were so overwhelmingly happy. Assuming the suggestions in general are not distractingly wrong, then 8% of code a
by pavlov 3y ago
I think the 8% number better explains why users were so overwhelmingly happy. Assuming the suggestions in general are not distractingly wrong, then 8% of code automatically written is a decent amount of time saved researching solutions.
- layer8 3y agoBut only 22% are accepted for those 8%, which means that the 78% code suggestions that are not accepted correspond to an equivalent of over 28% of all code written. Not sure that having to spend the time evaluating an additional 28% of code in vain amounts to an overall win. Though I guess the success rates when using Stack Overflow aren’t too dissimilar.
- shostack 3y agoWhat it doesn't tell us though is how useful the rejected recommendations were. Meaning, how many rejected solutions were sufficient to give the engineer enough context to turn a 30m task into a 5m task because they generated a recommendation, got an idea, rejected it, and rewrote it more efficiently or more correctly? There's a lot of "devil in the details" likely buried in here.
- visarga 3y agoInteresting that 91% find it useful but only 8% of the code is generated by LLM. This is even with a LLM tuned on the internal codebase. This will give a mild boost but not replace anyone.