3 ms·

Same goes for John though, doesn't it?

• SoftTalker 3y ago
  Yes, it does, but John is the subject matter expert and is at least in theory able to verify that the results are correct.
  • jameshart 3y ago
    No, John is four years from retirement and he sometimes doesn’t sleep that well. He has a lot on his mind what with the layoffs recently. Maybe you got him on a good day though.
• marcosdumay 3y ago
  A random John has a much better track record of getting the result right instead of making it all up. And you can tell John to be extra careful (in fact, if he has a bit of experience, you won't have to tell this either) and he will be. While if you tell the LLM to be extra careful, it may track intermediate steps better or just make some other answer up. And yeah, this is very likely intrinsic to the architecture we use on our LLMs.
• MrJohz 3y ago
  This is true, and a good reason why automation is so valuable. It's not just the speed, but also the reproducibility. If the script goes wrong, it will go wrong in a predictable way that can be investigated and debugged. If John goes wrong, he will go wrong sporadically and without any clear reason - maybe he was tired one day, or maybe he got an important phone call in the middle of the task another. With AI (or at least the versions of it that we're looking at today) we get the speed of a script with the "programmability" of a person, but the question still remains what sort of failure mode we can expect - predictable like a script, or unpredictable like a person. Right now, it's looking like LLM failures tend to be fairly unpredictable (in the sense that a prompt can give a good response one day and a bad one the next based on difficult-to-explain factors).