3 ms·

Useless? Or not useful enough? These anecdotes represent a bad taste that may linger for a long time, even while the actual performance of LLMs moves from curi

Useless? Or not useful enough? These anecdotes represent a bad taste that may linger for a long time, even while the actual performance of LLMs moves from curiosity to utility to indispensability. I’m allergic to FOMO-based thinking but I am convinced we’re already at the “useful” stage for many use cases, when combined with human judgment.

jprete 3y ago
One of the things LLMs can't imitate is having serious stake in the game, and therefore responsibility for the outcome. The PM has a stake - if they get this stuff wrong frequently, their job is at risk. The LLM provider is probably not getting paid much for the work - nothing, if it's a freely available model - and since the fine print says "do not rely on factual accuracy", the provider will claim no responsibility for its mistakes. (And the fine print will continue to say that for the foreseeable future, because even though LLMs might get more accurate, the providers will never want liability for errors unless the user is paying through the nose.)
- shermantanktop 3y ago
  If the PM in question copy-pastes LLM output into a status report, they won't last long. If they read it, realize the LLM output missed a key point, disagree with the emphasis in a section, and failed to mention an underlying trend or concern---at that point the PM is engaging their own brain, fixing the issues, but is likely way faster to get to a good place. At that point the LLM is acting like a semi-competent intern.
freitzkriesler2 3y ago
Useless as in it harped upon information everyone was already aware of and completely ignored the mitigation strategies that were discussed. The PM ended up going back and listening to the recording because the strategies were what was the most important part of the discussion.

3 ms·

Useless? Or not useful enough? These anecdotes represent a bad taste that may linger for a long time, even while the actual performance of LLMs moves from curi

by shermantanktop 3y ago

jprete 3y ago
One of the things LLMs can't imitate is having serious stake in the game, and therefore responsibility for the outcome. The PM has a stake - if they get this stuff wrong frequently, their job is at risk. The LLM provider is probably not getting paid much for the work - nothing, if it's a freely available model - and since the fine print says "do not rely on factual accuracy", the provider will claim no responsibility for its mistakes. (And the fine print will continue to say that for the foreseeable future, because even though LLMs might get more accurate, the providers will never want liability for errors unless the user is paying through the nose.)
- shermantanktop 3y ago
  If the PM in question copy-pastes LLM output into a status report, they won't last long. If they read it, realize the LLM output missed a key point, disagree with the emphasis in a section, and failed to mention an underlying trend or concern---at that point the PM is engaging their own brain, fixing the issues, but is likely way faster to get to a good place. At that point the LLM is acting like a semi-competent intern.
freitzkriesler2 3y ago
Useless as in it harped upon information everyone was already aware of and completely ignored the mitigation strategies that were discussed. The PM ended up going back and listening to the recording because the strategies were what was the most important part of the discussion.