9 ms·

I’ve been working in the “real world business processes that companies are trying to AI-ify” realm for quite a while now. Pharma, cyber security, oil and gas pr

I’ve been working in the “real world business processes that companies are trying to AI-ify” realm for quite a while now. Pharma, cyber security, oil and gas production, etc. This article doesn’t mention a really, really straightforward factor for why AI hasn’t invaded these domains despite billions of dollars being dumped into them. An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so. 99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong.

syllogism 6y ago
In those sorts of domains the best pitch for AI is as a failsafe. The model and the human probably make different errors, often a human will make mistakes due to simple inattention. This lets you substitute the model for some other process controls that you'd need to maintain 100% accuracy, e.g. instead of having the work reviewed by a second person, you can have model + person. In lots of business contexts, probably most, reducing variance is much more valuable than reducing mean expenses. Variance can halt downstream production, so the loss can be some huge amount of opportunity. And variance propagates through a supply chain, so your customers will hate variance in your output, as it may mean they have to ship the variance forward, which their customers hate. Plus if you allow your supply chain to get away with inconsistency, they can start to rob you with lower average quality and it will take you time to notice. If a company has been bothering to do some process manually and they haven't outsourced it to the cheapest humans possible, then they care more about low variance than low cost. Pitching these businesses a solution that lowers cost at the expense of unknown high variance is very unattractive. Instead, you want to tell them "I can reduce your variance even further! Here's what that would cost".
- lqet 6y ago
  > instead of having the work reviewed by a second person, you can have model + person But the argument still applies then - if only one incident occurs where a catastrophic mistake was not spotted by the model failsafe, and if later investigations show that the mistake could've been easily spotted by a second human, a human will be installed as a failsafe for the failsafe. I think the general problem is the following: if a human makes a grand mistake, it can usually be attributed to a temporal lack of care, or just random bad luck, or happened because the person was having a bad day, or... it is also generally understood that making a grand mistake will be such a shock for the person responsible that this person will most likely never make the same mistake again. On the other hand, if a machine makes a grand mistake, the intuition of the general public, trained by centuries of experience with techology, is that this machine will make the same mistake again and again and again, when prompted with the same input. If the model is not designed to learn from mistakes in production, this will of course actually be true.
  - MaxBarraclough 6y ago
    > But the argument still applies then [...] a human will be installed as a failsafe for the failsafe. Surely this will depend on the cost of failure, and the cost of the human failsafe. Spellcheck in an email client helps prevent the minor embarrassment of typos and spelling errors in emails, and few emails are so consequential that it's worth having them carefully manually vetted.
    - lqet 6y ago
      Of course, this is why I wrote "catastrophic" above. I also would not call an email spellchecker a business use case. Regarding spellcheckers per se: I have worked with book publishing companies in the last few years, and I can assure you all major book publishers employ real humans for the final spellchecking before a book goes into print.
  - fauigerzigerk 6y ago
    But the parent's point is that "the model and the human probably make different errors". That is frequently true. Take the job of a lifeguard for instance. A single mistake can be catastrophic, and yet we know that people have trouble staying completely focused for hours on end. AIs have no trouble staying focused and they can be trained to spot drowing swimmers pretty well even on a crowded beach. Having a human lifeguard plus an AI that alerts the lifeguard when it spots something suspicious could lead to better outcomes than employing two lifeguards.
- woeirua 6y ago
  Except that in most practical scenarios, most models make basic mistakes that no human would ever make. In most cases, people have a very low tolerance for errors that humans would not make.
ChuckNorris89 6y ago
Huh, N26, a major online bank in Europe is famous for some of its customers getting their accounts blocked every time they tweak their ML model and yet is doing great financially dispite the shitstorm it generates each time. It's not like Google blocking your email or YouTube account, we're taking about your friggin bank account here. I don't know how they're still in business and growing with such a process in place.
- ethanbond 6y ago
  I’ve found that financial institutions have a disproportionately large appetite for this. This is frankly because they view their own mission criticality as somewhat low — lower even than their customers perceive it. Undoing bad automation in finance often means reversing a very cheap edit on some database and pissing off a customer who’s so powerless that it won’t affect your business anyway.
- morelisp 6y ago
  > N26... is doing great financially. Are they?
- gridlockd 6y ago
  You are conflating growth with financial success. N26 is still losing money. A company that gives away candy can easily achieve massive growth. I doubt its customers are generally aware of the AI issue, or if they are, they must assume it is not out of the ordinary.
- bsaul 6y ago
  Not working directly in the field, but i believe ML for fraud detection is very common nowadays..
  - saiya-jin 6y ago
    Yeah but usually the bank just buys package solution with some customization for accessing the data & reporting, so they don't have to keep up the army of very narrowly specialized folks catching up with ever changing laws etc.
amcoastal 6y ago
Do you do any work with uncertainty estimation with neural networks? There are many different ways to estimate uncertainty depending on your application, such as ensemble-training or dropout during testing to produce ranges of predictions that you can then use to get boundable error measurements.
- colinmorelli 6y ago
  That sort of misses the point. The uncertainty estimation is, itself, dependent on and a direct result of the quality of the training data you give it and that training data's representation of reality. And even if the inputs are great, it's hard to understand what we can do with a 99% confidence prediction. The next great unlock that society is waiting for in the AI realm applies to industries where failure is not an easily acceptable outcome. Take self-driving cars for example. Even if we can objectively prove that current AI models can drive on current roads in current conditions with a lower fatality rate than humans (this is debatable anyway, but let's assume) - what do we do when it knows it's not confident? If we assume the driver hasn't been paying attention during the ride so far, and a scenario comes up that the AI is uncertain about...the human now has likely mere seconds (at most) to capture their surroundings, analyze the risk, and take corrective action. If we assume the driver has been paying attention during the ride, then what was the point of the AI? Moreover: what if it thought it was confident, but it was still wrong. Who do we blame? How do we mitigate future instances of it? What were once societal problems we could blame on a fallible humans are now obscure technology problems we can't introspect. That's a scary place for a lot of people to be. Basically, we've reached the level of AI where it can be used as a backup to humans and protect us from royally fucking something up. But the next big unlock will come when it can be the primary actor. It's hard to imagine how we'll get there without the AI actually understanding what it's processing in some real way. Edit: rephrased the 3rd paragraph.
  - amcoastal 6y ago
    I agree, the question of how to deal with high uncertainties is definitely task dependent. In your example of a self-driving car the answer is not clear. However, for many other tasks its entirely reasonable to have the AI make decisions and then pop it out to human review upon high uncertainty, or just not act on it. In tasks like this AI can be the primary task-doer with the human as the backup.
cameldrv 6y ago
Humans are hardly ever 99% accurate instantaneously on classification problems either. Humans have the ability to know when they should collect more information though, and can often perform some sort of hedging action when they are unsure.
- libertine 6y ago
  But isn't also the case that humans understand the consequences and the depth of multi-variable decisions? For example, Amazon Seller Central, Google, Youtube, can "outsource" their customer service to AI, because they are pretty much the only players, so customers have to suck it and deal with the frustration of a terrible experience by not getting help and not talking with a human. With any other business if they get automated replies that don't solve a customer issue and they are unable to talk with a human, 99% of them say "fuck it" and go somewhere else. This is one small realm of the relationship of AI and businesses. Then you have employees, suppliers, supply chains, finances, internal processes, so many multi-variable, fragile and nuanced systems that I doubt if they're not developed in-house, they'll probably do more damages than solve problems.
- simonh 6y ago
  This is interesting to reason about because it's may even be true that the human and AI error rates could be the same. In fact the human error rate might even we worse, but the kinds of errors and their impact makes a big difference. I can only speculate, but while it's true humans can fail when following a process, it's also true they can sometimes spot a potential failure even then there is no process to prevent it. They can come up with new processes, or ways to improve existing ones. They can also account for their actions, and managers can account for the activities of their team. All of this builds trust that the process can be improved in ways that can be well understood. People are also scalable. You can implement controls like four-eyes on changes and critical metrics, so you're less exposed to one person's idiosyncrasies. It's hard to do that with AI.
  - cameldrv 6y ago
    I agree. Humans are interactive agents, they have broader knowledge about the world, and they can (to a degree) diagnose their performance and compensate. Think of a DL powered, visually guided robot vs. a human on a production line. One task might be to do QC inspection at the end of the line. Suppose something gets on the camera lens. In general, the DL system will keep chugging along and the accuracy will degrade. The human will notice this and clean his glasses. If he sees something ambiguous as it passes on the line, he might give it a bit more attention or adjust the angle he's looking at it from. If he sees a series of the same anomalies, he will notice a pattern. Perhaps one of the machines up the line from him has started to introduce a new type of defect. Suppose in assembly a worker has a sore muscle. He might adapt his motions to compensate, slow down, go to the doctor, take some pain pills, or take a day off. Unless programmed or trained to detect this, a robot will keep driving its failing motor harder until it breaks.
amelius 6y ago
> 99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong. Like with self-driving cars?
- SamuelAdams 6y ago
  Yep that's a great example. For other businesses: Pharmacy - adding 1% more of a chemical compound to a pill could be lethal, but not otherwise detected for 3-12 months (pills sit on shelves for a long time). Figuring out why that happened will be very difficult. Manufacturing - molding plastic parts requires very specific mixes of chemicals. Too much of one and it becomes brittle under certain temperatures, too little and it warps down under the warranty period. So if you manufacture the mold of a baby car seat, and you add or subtract 1% of a chemical, that car seat could break / shatter when it's involved in a collision. Terribly large lawsuits would occur because some C-level someone adopted a process to save headcount and reduce oversight. Business processes are a lot like software - the first 80% takes 20% of the time and the remaining 20% takes 80% of the time.
tgb 6y ago
AI for pharma seems to be targeted at things like drug discovery, where yes every output from the algorithm has to be carefully validated. But that's already the case for human-generated ideas. So I don't think your argument makes sense in this context.
- ethanbond 6y ago
  Well it’s a bit simplistic to say that it’s used “for drug discovery.” A particular application might be doing NLP on e.g. clinicians’ notes or on scientific journals. There are plenty of places where human validation shouldn’t be necessary, but is.
x3blah 6y ago
Why dump billions of dollars then? Nowhere else to spend it? Effective marketing?[1] Is no one asking this question? "... and critically: there's no one to hold responsible for getting it wrong." Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.[2] 1. "What's more, artificial research teams lack an awareness of the specific business processes and tasks that could be automated in the first place. Researchers would need to develop an intuition of the business processes involved. We haven't seen this happen in too many areas." 2. Including the ones who designed the "AI" system.
- colinmorelli 6y ago
  > Why dump billions of dollars then? Nowhere else to spend it? Effective marketing? Is no one asking this question? Because whoever does achieve the next unlock - should it happen - will receive an unimaginably large windfall. This is the classic intent of venture capital. In fact, I'd suggest that AI is actually one industry where VC is doing what it does best: taking extremely risky bets with a large potential upside. > Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability. Presently, this seems to be one of its large detractors. If I have an employee do something stupid, I can say that an employee did something stupid. People might wonder why they were allowed to do that stupid thing, and what we're going to do to prevent it from happening again, but the explanation of the source is satisfactory. We're fallible, and we understand the fallibility of others (generally speaking). AI is not that at all. If my automation does something stupid, I still have all the blame, and yet I have nowhere else to pass it off to. "We don't understand why our AI did this really stupid thing" is, frankly, not a satisfying response (nor should it be). Businesses employing AI certainly are not absolved of any form of accountability, and are arguably exposed to more of it (since they're not able to pass the blame on to another fallible human, and have to take direct accountability of a system they built but don't fully understand).
- Veedrac 6y ago
  You don't need to solve every problem to make AI worth the investment, just a large enough handful of billion-dollar problems.
Majromax 6y ago
> 99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong. AI also exposes the possibility of systemic error where humans would be stochastic. A human might only identify the right number of rentable units from a spreadsheet (to pick an example from this article) 97% of the time when an AI might do it 99% of the time, but even the same human will have a different 3% error on each day. The consequences of failure are more limited and more dilute. On the other hand, the AI may work perfectly right up until a holding company redesigns their data tables for the 100th time, whereupon it misreads every financial report with much more concentrated ill effect.
kgin 6y ago
The blame issue is huge. When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate. But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.
- afeller08 6y ago
  This. This is strictly more accurate than the above comment. It has nothing to do with accuracy or the repetitiveness of the mistake. It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes, because most companies are designed to absolve everyone at the top and blame everyone at the bottom. I've worked in an industry (prop trading) where the costs of mistakes were real, and it is the most heavily automated industry there is. I've also worked in another part of the finance industry (hedge funds) where mistakes are paid for in other people's money, and it's super obvious that managers there are way more interested in passing off blame than fixing problems relative to how interested they are in doing these things in prop trading, and there is correspondingly less automation. It's not about people in the population at large believing that machines are more prone to repeating mistakes than humans. Look at how many people are excited about the concept of self driving cars. It's about managers trying to convince themselves that this is true, so they can go on being unethical.
  - jdm2212 6y ago
    > It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes You can't fire a crappy AI and replace it with one with better judgement. If it does something weird or bad or inexplicable, you can't ask why it did that and tell it to how to be better in the future. Blame is a tool that organizations and people; you can't blame AI because you mostly can't teach AI.
SkyBelow 6y ago
>An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so. I think this is the core of the problem. 99% isn't good enough. Even 99.9% isn't good enough when we are talking acceptable accepted error margins. Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance. We see this with self driving cars. People's reactions to the machines is that the machines have to be far better than humans before humans will be okay with the risks involved. This also holds for financial aspects. Imagine your grocery store telling you that there is a X% chance of being double charged for an item and that is within acceptable error tolerance. Even if X is lower than the rate that human grocers accidentally double charge will people be okay with that as the planned error rate or will they demand perfection?
- vsareto 6y ago
  Even if you balance the double charging with a chance to get an item for free (or negative price, acting as a credit), humans will still complain when they get double charged and stay silent when they get the bonus. But of course there's rich people who don't even care they were double charged for something like groceries. Most of these issues aren't about how many 9's of reliability there are, but whether or not some person is accountable. AI is not itself accountable, only the person in charge of it. Many companies already give gift cards or whatever when a human mistake happens as a form of customer service, and there's not enough compensation around AI error rates to make them palatable.
- afeller08 6y ago
  >Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance. The veracity of this statement greatly depends on whose mistake you are saying it was. List of things by how hard they are to say (easiest to hardest): 1) Someone else messed up. We'll deal with him. 2) Our code is bad. We'll fix it. 3) I messed up. I'm sorry.
nova22033 6y ago
An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. Wouldn't this itself be error prone? The operator is, after all, human.
- ethanbond 6y ago
  Yes but you can fire them without dismantling your IT infrastructure.
ganstyles 6y ago
This is well put, and I've worked places where this has happened to customers. Our current domain is in augmenting humans with AI, which avoids the problem you're describing because it doesn't take humans out of the loop at all, but just makes them more efficient at what they were already doing by cutting out a lot of work. So jobs that would take an operator 30 minutes now take 10 because the AI has done most of the usual rote work.
openasocket 6y ago
From what I've seen of attempts to apply AI to the cyber security domain, a big impediment tends to be the lack of cohesion between the AI experts and the domain experts. There's often decades of work on the domain of study, and it's important that you use this information to inform your algorithms/ML models. I once worked with some ML researchers on trying to apply machine learning to some netflow data we had. These researchers were quite good at their field, but didn't know much about netflow data analysis. I remember them being all excited in a meeting because their models found some pattern in the data. It turns out they had discovered the difference between high ports and low ports...
lubesGordi 6y ago
Basically what it sounds like you're saying is that AI is non-deterministic, and you can't have systems like that running critical applications.
woeirua 6y ago
I wish I could upvote this twice. This is the exact problem preventing widespread adoption of "AI" throughout most businesses. A model that is 99% accurate is not good enough for most mundane business tasks because they will fail in ways that no human ever would. Additionally, when you consider how much it costs upfront to hire a team of engineers and data scientists, build a training dataset, develop the code, maintain it etc, it quickly becomes clear that except for the most costly processes internally there is no way that AI is going to be cheaper than hiring a bunch of people in India by the hour. Not to mention that the people in India can be retrained to do something in hours that would take your super specialized team of engineers months to reproduce with code. Let's also not forget that most business processes are bespoke to each corporation. Finding a single process that can be successfully targeted across multiple companies is hard. Some people like AWS and Google are trying with Textract and some of these other AI-as-a-service products, but they're not having a lot of success. They still fail all the time.
deleted 6y ago
[deleted]

syllogism 6y ago

In those sorts of domains the best pitch for AI is as a failsafe. The model and the human probably make different errors, often a human will make mistakes due to simple inattention. This lets you substitute the model for some other process controls that you'd need to maintain 100% accuracy, e.g. instead of having the work reviewed by a second person, you can have model + person. In lots of business contexts, probably most, reducing variance is much more valuable than reducing mean expenses. Variance can halt downstream production, so the loss can be some huge amount of opportunity. And variance propagates through a supply chain, so your customers will hate variance in your output, as it may mean they have to ship the variance forward, which their customers hate. Plus if you allow your supply chain to get away with inconsistency, they can start to rob you with lower average quality and it will take you time to notice. If a company has been bothering to do some process manually and they haven't outsourced it to the cheapest humans possible, then they care more about low variance than low cost. Pitching these businesses a solution that lowers cost at the expense of unknown high variance is very unattractive. Instead, you want to tell them "I can reduce your variance even further! Here's what that would cost".

lqet 6y ago
> instead of having the work reviewed by a second person, you can have model + person But the argument still applies then - if only one incident occurs where a catastrophic mistake was not spotted by the model failsafe, and if later investigations show that the mistake could've been easily spotted by a second human, a human will be installed as a failsafe for the failsafe. I think the general problem is the following: if a human makes a grand mistake, it can usually be attributed to a temporal lack of care, or just random bad luck, or happened because the person was having a bad day, or... it is also generally understood that making a grand mistake will be such a shock for the person responsible that this person will most likely never make the same mistake again. On the other hand, if a machine makes a grand mistake, the intuition of the general public, trained by centuries of experience with techology, is that this machine will make the same mistake again and again and again, when prompted with the same input. If the model is not designed to learn from mistakes in production, this will of course actually be true.
- MaxBarraclough 6y ago
  > But the argument still applies then [...] a human will be installed as a failsafe for the failsafe. Surely this will depend on the cost of failure, and the cost of the human failsafe. Spellcheck in an email client helps prevent the minor embarrassment of typos and spelling errors in emails, and few emails are so consequential that it's worth having them carefully manually vetted.
  - lqet 6y ago
    Of course, this is why I wrote "catastrophic" above. I also would not call an email spellchecker a business use case. Regarding spellcheckers per se: I have worked with book publishing companies in the last few years, and I can assure you all major book publishers employ real humans for the final spellchecking before a book goes into print.
- fauigerzigerk 6y ago
  But the parent's point is that "the model and the human probably make different errors". That is frequently true. Take the job of a lifeguard for instance. A single mistake can be catastrophic, and yet we know that people have trouble staying completely focused for hours on end. AIs have no trouble staying focused and they can be trained to spot drowing swimmers pretty well even on a crowded beach. Having a human lifeguard plus an AI that alerts the lifeguard when it spots something suspicious could lead to better outcomes than employing two lifeguards.
woeirua 6y ago
Except that in most practical scenarios, most models make basic mistakes that no human would ever make. In most cases, people have a very low tolerance for errors that humans would not make.

ChuckNorris89 6y ago

Huh, N26, a major online bank in Europe is famous for some of its customers getting their accounts blocked every time they tweak their ML model and yet is doing great financially dispite the shitstorm it generates each time. It's not like Google blocking your email or YouTube account, we're taking about your friggin bank account here. I don't know how they're still in business and growing with such a process in place.

ethanbond 6y ago
I’ve found that financial institutions have a disproportionately large appetite for this. This is frankly because they view their own mission criticality as somewhat low — lower even than their customers perceive it. Undoing bad automation in finance often means reversing a very cheap edit on some database and pissing off a customer who’s so powerless that it won’t affect your business anyway.
morelisp 6y ago
> N26... is doing great financially. Are they?
gridlockd 6y ago
You are conflating growth with financial success. N26 is still losing money. A company that gives away candy can easily achieve massive growth. I doubt its customers are generally aware of the AI issue, or if they are, they must assume it is not out of the ordinary.
bsaul 6y ago
Not working directly in the field, but i believe ML for fraud detection is very common nowadays..
- saiya-jin 6y ago
  Yeah but usually the bank just buys package solution with some customization for accessing the data & reporting, so they don't have to keep up the army of very narrowly specialized folks catching up with ever changing laws etc.

amcoastal 6y ago

Do you do any work with uncertainty estimation with neural networks? There are many different ways to estimate uncertainty depending on your application, such as ensemble-training or dropout during testing to produce ranges of predictions that you can then use to get boundable error measurements.

colinmorelli 6y ago
That sort of misses the point. The uncertainty estimation is, itself, dependent on and a direct result of the quality of the training data you give it and that training data's representation of reality. And even if the inputs are great, it's hard to understand what we can do with a 99% confidence prediction. The next great unlock that society is waiting for in the AI realm applies to industries where failure is not an easily acceptable outcome. Take self-driving cars for example. Even if we can objectively prove that current AI models can drive on current roads in current conditions with a lower fatality rate than humans (this is debatable anyway, but let's assume) - what do we do when it knows it's not confident? If we assume the driver hasn't been paying attention during the ride so far, and a scenario comes up that the AI is uncertain about...the human now has likely mere seconds (at most) to capture their surroundings, analyze the risk, and take corrective action. If we assume the driver has been paying attention during the ride, then what was the point of the AI? Moreover: what if it thought it was confident, but it was still wrong. Who do we blame? How do we mitigate future instances of it? What were once societal problems we could blame on a fallible humans are now obscure technology problems we can't introspect. That's a scary place for a lot of people to be. Basically, we've reached the level of AI where it can be used as a backup to humans and protect us from royally fucking something up. But the next big unlock will come when it can be the primary actor. It's hard to imagine how we'll get there without the AI actually understanding what it's processing in some real way. Edit: rephrased the 3rd paragraph.
- amcoastal 6y ago
  I agree, the question of how to deal with high uncertainties is definitely task dependent. In your example of a self-driving car the answer is not clear. However, for many other tasks its entirely reasonable to have the AI make decisions and then pop it out to human review upon high uncertainty, or just not act on it. In tasks like this AI can be the primary task-doer with the human as the backup.

cameldrv 6y ago

Humans are hardly ever 99% accurate instantaneously on classification problems either. Humans have the ability to know when they should collect more information though, and can often perform some sort of hedging action when they are unsure.

libertine 6y ago
But isn't also the case that humans understand the consequences and the depth of multi-variable decisions? For example, Amazon Seller Central, Google, Youtube, can "outsource" their customer service to AI, because they are pretty much the only players, so customers have to suck it and deal with the frustration of a terrible experience by not getting help and not talking with a human. With any other business if they get automated replies that don't solve a customer issue and they are unable to talk with a human, 99% of them say "fuck it" and go somewhere else. This is one small realm of the relationship of AI and businesses. Then you have employees, suppliers, supply chains, finances, internal processes, so many multi-variable, fragile and nuanced systems that I doubt if they're not developed in-house, they'll probably do more damages than solve problems.
simonh 6y ago
This is interesting to reason about because it's may even be true that the human and AI error rates could be the same. In fact the human error rate might even we worse, but the kinds of errors and their impact makes a big difference. I can only speculate, but while it's true humans can fail when following a process, it's also true they can sometimes spot a potential failure even then there is no process to prevent it. They can come up with new processes, or ways to improve existing ones. They can also account for their actions, and managers can account for the activities of their team. All of this builds trust that the process can be improved in ways that can be well understood. People are also scalable. You can implement controls like four-eyes on changes and critical metrics, so you're less exposed to one person's idiosyncrasies. It's hard to do that with AI.
- cameldrv 6y ago
  I agree. Humans are interactive agents, they have broader knowledge about the world, and they can (to a degree) diagnose their performance and compensate. Think of a DL powered, visually guided robot vs. a human on a production line. One task might be to do QC inspection at the end of the line. Suppose something gets on the camera lens. In general, the DL system will keep chugging along and the accuracy will degrade. The human will notice this and clean his glasses. If he sees something ambiguous as it passes on the line, he might give it a bit more attention or adjust the angle he's looking at it from. If he sees a series of the same anomalies, he will notice a pattern. Perhaps one of the machines up the line from him has started to introduce a new type of defect. Suppose in assembly a worker has a sore muscle. He might adapt his motions to compensate, slow down, go to the doctor, take some pain pills, or take a day off. Unless programmed or trained to detect this, a robot will keep driving its failing motor harder until it breaks.

amelius 6y ago

SamuelAdams 6y ago
Yep that's a great example. For other businesses: Pharmacy - adding 1% more of a chemical compound to a pill could be lethal, but not otherwise detected for 3-12 months (pills sit on shelves for a long time). Figuring out why that happened will be very difficult. Manufacturing - molding plastic parts requires very specific mixes of chemicals. Too much of one and it becomes brittle under certain temperatures, too little and it warps down under the warranty period. So if you manufacture the mold of a baby car seat, and you add or subtract 1% of a chemical, that car seat could break / shatter when it's involved in a collision. Terribly large lawsuits would occur because some C-level someone adopted a process to save headcount and reduce oversight. Business processes are a lot like software - the first 80% takes 20% of the time and the remaining 20% takes 80% of the time.

tgb 6y ago

AI for pharma seems to be targeted at things like drug discovery, where yes every output from the algorithm has to be carefully validated. But that's already the case for human-generated ideas. So I don't think your argument makes sense in this context.

ethanbond 6y ago
Well it’s a bit simplistic to say that it’s used “for drug discovery.” A particular application might be doing NLP on e.g. clinicians’ notes or on scientific journals. There are plenty of places where human validation shouldn’t be necessary, but is.

x3blah 6y ago

Why dump billions of dollars then? Nowhere else to spend it? Effective marketing?[1] Is no one asking this question? "... and critically: there's no one to hold responsible for getting it wrong." Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.[2] 1. "What's more, artificial research teams lack an awareness of the specific business processes and tasks that could be automated in the first place. Researchers would need to develop an intuition of the business processes involved. We haven't seen this happen in too many areas." 2. Including the ones who designed the "AI" system.

colinmorelli 6y ago
> Why dump billions of dollars then? Nowhere else to spend it? Effective marketing? Is no one asking this question? Because whoever does achieve the next unlock - should it happen - will receive an unimaginably large windfall. This is the classic intent of venture capital. In fact, I'd suggest that AI is actually one industry where VC is doing what it does best: taking extremely risky bets with a large potential upside. > Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability. Presently, this seems to be one of its large detractors. If I have an employee do something stupid, I can say that an employee did something stupid. People might wonder why they were allowed to do that stupid thing, and what we're going to do to prevent it from happening again, but the explanation of the source is satisfactory. We're fallible, and we understand the fallibility of others (generally speaking). AI is not that at all. If my automation does something stupid, I still have all the blame, and yet I have nowhere else to pass it off to. "We don't understand why our AI did this really stupid thing" is, frankly, not a satisfying response (nor should it be). Businesses employing AI certainly are not absolved of any form of accountability, and are arguably exposed to more of it (since they're not able to pass the blame on to another fallible human, and have to take direct accountability of a system they built but don't fully understand).
Veedrac 6y ago
You don't need to solve every problem to make AI worth the investment, just a large enough handful of billion-dollar problems.

Majromax 6y ago

> 99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong. AI also exposes the possibility of systemic error where humans would be stochastic. A human might only identify the right number of rentable units from a spreadsheet (to pick an example from this article) 97% of the time when an AI might do it 99% of the time, but even the same human will have a different 3% error on each day. The consequences of failure are more limited and more dilute. On the other hand, the AI may work perfectly right up until a holding company redesigns their data tables for the 100th time, whereupon it misreads every financial report with much more concentrated ill effect.

kgin 6y ago

The blame issue is huge. When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate. But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.

afeller08 6y ago
This. This is strictly more accurate than the above comment. It has nothing to do with accuracy or the repetitiveness of the mistake. It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes, because most companies are designed to absolve everyone at the top and blame everyone at the bottom. I've worked in an industry (prop trading) where the costs of mistakes were real, and it is the most heavily automated industry there is. I've also worked in another part of the finance industry (hedge funds) where mistakes are paid for in other people's money, and it's super obvious that managers there are way more interested in passing off blame than fixing problems relative to how interested they are in doing these things in prop trading, and there is correspondingly less automation. It's not about people in the population at large believing that machines are more prone to repeating mistakes than humans. Look at how many people are excited about the concept of self driving cars. It's about managers trying to convince themselves that this is true, so they can go on being unethical.
- jdm2212 6y ago
  > It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes You can't fire a crappy AI and replace it with one with better judgement. If it does something weird or bad or inexplicable, you can't ask why it did that and tell it to how to be better in the future. Blame is a tool that organizations and people; you can't blame AI because you mostly can't teach AI.

SkyBelow 6y ago

>An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so. I think this is the core of the problem. 99% isn't good enough. Even 99.9% isn't good enough when we are talking acceptable accepted error margins. Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance. We see this with self driving cars. People's reactions to the machines is that the machines have to be far better than humans before humans will be okay with the risks involved. This also holds for financial aspects. Imagine your grocery store telling you that there is a X% chance of being double charged for an item and that is within acceptable error tolerance. Even if X is lower than the rate that human grocers accidentally double charge will people be okay with that as the planned error rate or will they demand perfection?

vsareto 6y ago
Even if you balance the double charging with a chance to get an item for free (or negative price, acting as a credit), humans will still complain when they get double charged and stay silent when they get the bonus. But of course there's rich people who don't even care they were double charged for something like groceries. Most of these issues aren't about how many 9's of reliability there are, but whether or not some person is accountable. AI is not itself accountable, only the person in charge of it. Many companies already give gift cards or whatever when a human mistake happens as a form of customer service, and there's not enough compensation around AI error rates to make them palatable.
afeller08 6y ago
>Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance. The veracity of this statement greatly depends on whose mistake you are saying it was. List of things by how hard they are to say (easiest to hardest): 1) Someone else messed up. We'll deal with him. 2) Our code is bad. We'll fix it. 3) I messed up. I'm sorry.

nova22033 6y ago

An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. Wouldn't this itself be error prone? The operator is, after all, human.

ethanbond 6y ago
Yes but you can fire them without dismantling your IT infrastructure.

ganstyles 6y ago

This is well put, and I've worked places where this has happened to customers. Our current domain is in augmenting humans with AI, which avoids the problem you're describing because it doesn't take humans out of the loop at all, but just makes them more efficient at what they were already doing by cutting out a lot of work. So jobs that would take an operator 30 minutes now take 10 because the AI has done most of the usual rote work.

openasocket 6y ago

From what I've seen of attempts to apply AI to the cyber security domain, a big impediment tends to be the lack of cohesion between the AI experts and the domain experts. There's often decades of work on the domain of study, and it's important that you use this information to inform your algorithms/ML models. I once worked with some ML researchers on trying to apply machine learning to some netflow data we had. These researchers were quite good at their field, but didn't know much about netflow data analysis. I remember them being all excited in a meeting because their models found some pattern in the data. It turns out they had discovered the difference between high ports and low ports...

lubesGordi 6y ago

Basically what it sounds like you're saying is that AI is non-deterministic, and you can't have systems like that running critical applications.

woeirua 6y ago

I wish I could upvote this twice. This is the exact problem preventing widespread adoption of "AI" throughout most businesses. A model that is 99% accurate is not good enough for most mundane business tasks because they will fail in ways that no human ever would. Additionally, when you consider how much it costs upfront to hire a team of engineers and data scientists, build a training dataset, develop the code, maintain it etc, it quickly becomes clear that except for the most costly processes internally there is no way that AI is going to be cheaper than hiring a bunch of people in India by the hour. Not to mention that the people in India can be retrained to do something in hours that would take your super specialized team of engineers months to reproduce with code. Let's also not forget that most business processes are bespoke to each corporation. Finding a single process that can be successfully targeted across multiple companies is hard. Some people like AWS and Google are trying with Textract and some of these other AI-as-a-service products, but they're not having a lot of success. They still fail all the time.

deleted 6y ago

[deleted]