4 ms·
An LLM can certainly try to consume an extremely poorly specified bug report, in half English half the user's native language, then consume the entire codebase
by zeta0134 1y ago
An LLM can certainly try to consume an extremely poorly specified bug report, in half English half the user's native language, then consume the entire codebase and guess what the devil they're actually referring to. My guess is the humans are better at this, and I mostly speak from experience on two separate support floors that tried to add AI to that flow. It fails, miserably.
It's not really possible for an LLM to pick up on the hidden complexities of the app that real users and developers internalize through practice. Almost by definition, they're not documented! Users "just know" and thus there is no training data to ingest. I'd estimate though that the vast, vast majority of bugs I've been assigned originate from one of these vague reports by a client paying us enough to care.