5 ms·

> No one longed for a level of AI where you have to double check everything. This has basically been why it's a non-starter in a lot of (most?) business applic

by mjr00 2y ago

> No one longed for a level of AI where you have to double check everything. This has basically been why it's a non-starter in a lot of (most?) business applications. If your dishwasher failed to clean anything 20% of the time, would you rely on it? No, you'd just wash the dishes by hand, because you'd at least have a consistent result. That's been the result of AI experimentation I've seen: it works ~80% of the time, which sounds great... except there's surprisingly few tasks where a 20% fail rate is acceptable. Even "prompt engineering" your way to a 5% failure/inaccuracy rate is unacceptable for a fully automated solution. So now we're moving to workflows where AI generates stuff and a human double checks. Or the AI parses human text into a well-defined gRPC method with known behavior. Which can definitely be helpful, but is a far cry from the fantasized AI in sci-fi literature.

Nition 2y ago
One type of question that a 20%-failure-rate AI can still be very useful for is ones that are hard to answer but easy to verify. For example say you have a complex medical problem. It can be difficult to do a direct Internet search that covers the history and symptoms. If you ask AI though, it'll be able to give you some ideas for specific things to search. They might be wrong answers, but now you can easily search specific conditions and check them. Sort of P vs. NP for questions.
- skydhash 2y ago
  > For example say you have a complex medical problem. Or you go to a doctor instead of imagining answers.
  - Nition 2y ago
    Medical was just one example, replace with anything you like. As another example, you can give the AI a photo of something to have it name what that thing is. Then you can check the thing by its name on Google to see if it matches. Much easier than describing the thing (plant, tool, etc) to Google.
    - skydhash 2y ago
      Having the wrong information can be more detrimental than having no information at all. In the former case, confident actions will be take. In the latter case, the person will be tentative wich can reduce the area of effect of bad decisions. Imagine the lambda person confronted with this: sudo rm -rf / What is the better situation, having no understanding of what it does or believing that another action will take place?
      - Nition 1y ago
        The process I'm suggesting is: 1. You have a complex or vague question that you can't search easily via Google etc 2. You ask the AI and it converts that to concrete searchable suggestions (in this case "sudo rm -rf /") 3. You search "sudo rm -rf /" to check the answer. Step 3 is designed to (hopefully) catch this kind of problem.
  - ianbutler 2y ago
    You put too much faith in doctors. Pretty much every woman I know has been waived off for issues that turned serious later and even as a guy I have to do above average leg work to get them to care about anything.
    - satisfice 2y ago
      Doctors are still better than LLMs, by a lot.
      - ninetyninenine 2y ago
        To add to that real doctors have incentives which lead to malpractice. Malpractice is not a minor issue
      - Ancapistani 2y ago
        All the recent studies I’ve read actually show the opposite - that even models that are no longer considered useful are as good or better at diagnosis than the mean human physician.
  - bdangubic 2y ago
    literally the LAST place I would go (I am American)
dclowd9901 2y ago
It feels a bit like LLMs rely a lot on _us_ to be useful. Which is a big point to the author's article about how companies are trimming off staff for AI.
- re-thc 2y ago
  > how companies are trimming off staff for AI But they're not. That's just the excuse. The real truth is somewhere along pandemic over hire and bad / unstable economy.
  - Terr_ 2y ago
    Also attempts to influence investors/stock-price. https://newrepublic.com/article/178812/big-tech-loves-lay-offs https://newrepublic.com/article/178812/big-tech-loves-lay-of...
  - dclowd9901 2y ago
    We've frozen hiring (despite already being under staffed) and our leadership has largely pointed to advances in AI as being accelerative to the point that we shouldn't need more bodies to be more productive. Granted it's just a personal anecdote but it still affects hundreds of people that otherwise would have been hired by us. What reason would they have to lie about that to us?

by mjr00 2y ago

Nition 2y ago

One type of question that a 20%-failure-rate AI can still be very useful for is ones that are hard to answer but easy to verify. For example say you have a complex medical problem. It can be difficult to do a direct Internet search that covers the history and symptoms. If you ask AI though, it'll be able to give you some ideas for specific things to search. They might be wrong answers, but now you can easily search specific conditions and check them. Sort of P vs. NP for questions.

skydhash 2y ago
> For example say you have a complex medical problem. Or you go to a doctor instead of imagining answers.
- Nition 2y ago
  Medical was just one example, replace with anything you like. As another example, you can give the AI a photo of something to have it name what that thing is. Then you can check the thing by its name on Google to see if it matches. Much easier than describing the thing (plant, tool, etc) to Google.
  - skydhash 2y ago
    Having the wrong information can be more detrimental than having no information at all. In the former case, confident actions will be take. In the latter case, the person will be tentative wich can reduce the area of effect of bad decisions. Imagine the lambda person confronted with this: sudo rm -rf / What is the better situation, having no understanding of what it does or believing that another action will take place?
    - Nition 1y ago
      The process I'm suggesting is: 1. You have a complex or vague question that you can't search easily via Google etc 2. You ask the AI and it converts that to concrete searchable suggestions (in this case "sudo rm -rf /") 3. You search "sudo rm -rf /" to check the answer. Step 3 is designed to (hopefully) catch this kind of problem.
- ianbutler 2y ago
  You put too much faith in doctors. Pretty much every woman I know has been waived off for issues that turned serious later and even as a guy I have to do above average leg work to get them to care about anything.
  - satisfice 2y ago
    Doctors are still better than LLMs, by a lot.
    - ninetyninenine 2y ago
      To add to that real doctors have incentives which lead to malpractice. Malpractice is not a minor issue
    - Ancapistani 2y ago
      All the recent studies I’ve read actually show the opposite - that even models that are no longer considered useful are as good or better at diagnosis than the mean human physician.
- bdangubic 2y ago
  literally the LAST place I would go (I am American)

dclowd9901 2y ago

It feels a bit like LLMs rely a lot on _us_ to be useful. Which is a big point to the author's article about how companies are trimming off staff for AI.

re-thc 2y ago
> how companies are trimming off staff for AI But they're not. That's just the excuse. The real truth is somewhere along pandemic over hire and bad / unstable economy.
- Terr_ 2y ago
  Also attempts to influence investors/stock-price. https://newrepublic.com/article/178812/big-tech-loves-lay-offs https://newrepublic.com/article/178812/big-tech-loves-lay-of...
- dclowd9901 2y ago
  We've frozen hiring (despite already being under staffed) and our leadership has largely pointed to advances in AI as being accelerative to the point that we shouldn't need more bodies to be more productive. Granted it's just a personal anecdote but it still affects hundreds of people that otherwise would have been hired by us. What reason would they have to lie about that to us?