7 ms·

I'm going to go against the flow of most comment here and say that it's not always business misunderstanding AI. Bad labeled data and unclear goals/expectations

I'm going to go against the flow of most comment here and say that it's not always business misunderstanding AI. Bad labeled data and unclear goals/expectations sure, but the latter one should be identifiable by a good ML/Data scientist, if you have any insight to what you can actually deliver. But most ML/Data Science people have no proper understanding of AI/ML, and when just traditional "coding" can solve the problem rather than throwing fancy statistical models and buzzword. I'm not in business nor AI/ML, but in Robotics. And as a person in Robotics it's always the same experience working with AI/ML engineers: They first say they require large amount of data, then give great promises. (but never specific metrics, except maybe a percentage of success) Then they deliver a module that fails outside the perfect scope of deployment(works only in the lab at 1pm). This is ofc never specified in the delivery. Also crucially, it does not give a good indication of failure. The amount of ad-hoc you need to add after the thing is delivered is just staggering. On top of this reality, most ML/Data science peoples response to this entire process is to point and blame the data, or the "well you guys is expecting too much from this!" when they had ample time to outline the scope, limitation and requirement before they even started collecting the data.

lmilcin 6y ago
I would add "AI" produces results that are hard to understand and work with. When you do "traditional coding" you are getting results that are easier to work with because you understand where they came form. Then it is easier to modify the model and understand the outcome. Your background is robotics. As a robotics example, training AI model to keep bipedal robot upright might be possible but having proper control model that came from control theory, identifying the system, etc. will mean you can actually calculate the envelope of the model, the limitations, etc. and that the model is not likely to surprise you in catastrophic ways. It is a case where AI would be a poor replacement for traditional modelling.
- astrophysician 6y ago
  I mean, part of the problem is that no one can seem to agree about what "AI" even is anymore; you described a control theory model that "calculate[s] the envelope" which I'm interpreting to mean some sort of confidence interval/posterior, but to me that's exactly what you should be doing whenever possible in an "AI" model, as in, control theory (it sounds like with Gaussian Processes or something) is the right "AI" approach to that problem. To me, linear/logistic regression is AI, as are neural nets, as are pretty much any statistical model. These days "AI" seems to be roughly interpreted as "whatever a DS person deems as a solution to our business problem" at least in my very limited experience. It's all a bit silly since AI has come to mean so many different things that may not fit the traditional definition of AI from e.g. the 90's, but as long as people aren't pidgeon-holeing themselves into particular methods, I'm happy to go along with slapping an "AI" label on pretty much anything.
  - sobellian 6y ago
    I took envelope to mean the operating conditions under which the control system is stable. For many control systems this isn't really a confidence interval except insofar as the parameters of the control system are imperfectly known. For as long as I can remember, AI has been a synonym for "hard CS problem" or even just "hard math problem." It has more use as a marketing term than as a technical term with precise meaning.
    - lmilcin 6y ago
      "I took envelope to mean the operating conditions under which the control system is stable." Yes, that's exactly what I meant. I kind of borrowed the term from aviation where this is used to describe conditions under which the behavior of the aircraft is known and predictable. Doesn't mean it is not possible to fly outside the envelope, you just don't get any guarantees.
    - astrophysician 6y ago
      > I took envelope to mean the operating conditions under which the control system is stable. For many control systems this isn't really a confidence interval except insofar as the parameters of the control system are imperfectly known. Ah gotcha, totally agree. > It has more use as a marketing term than as a technical term with precise meaning. yea exactly; not a problem for me that this is the case (buzzwords are just a fact of life), but really what should be happening underneath any "AI" solution is finding and implementing the right solution to the problem. If the right solution is classical control theory or linear models, but the business leaders insist on deep learning because to them that's what AI means, that's where there's a problem.
    - deleted 6y ago
      [deleted]
  - numpad0 6y ago
    I’m confident* that “AI” these days means DNN that takes n-dim input in n>2. Anything else is “contemporary“ or “conventional” maths or theories. * Read “not at all”
- NalNezumi 6y ago
  >I would add "AI" produces results that are hard to understand and work with. Sorry but no, this rhetoric is often used as an easy escape of responsibility but I don't buy it. Sure, if you're talking about the neural nets used in Google translate or YouTube recommendation system, or any big tech company domain of problem that is based on vast amount of data with extreme size of state space and variability - fine. But we are not talking about interpretability of the result but the reliability of it, and in robotics domain you rarely have to tackle that size of state space, but rather a highly noisy one. Let's take your example of low level control of bipedal robot. At whatever stage of control/manipulation you're talking about, you always have to consider that the input will have some noise, and how to handle it. This is the reliability; you know how to(or how much) of the input noise you can handle, and how much you will leave (or amplify) to the rest of the system. Based on this you also know the limit of the system, so you can also get an indication of failure. You can make AI/Neural Network to consider the noise and also give an indication of failure, but you have to think about that from the data gathering step! Most control algorithms have this: "what input noise can I handle" question answered; The same requirement should be set to the data. I assume you thought about Reinforcement Learning of sorts when you thought about control, but what most people don't realize is that those algorithms works only because you can use high precision sensor that reduce these noise level (combined with sensor fusion) to negligible level. But now you're relying on high precision, extremely expensive sensors. They almost always fail when you get out of the lab and have to suddenly consider cost. (and at this stage traditional control algorithm creeps back) Let's take a real example. On my original comment I wrote "work in lab at 1pm" and I weren't sarcastic. Apparently not only did the AI solution (in Computer Vision) I were given only work in the lab, but at the exact lightning condition under high noon at the lab, with all the ceiling lights on all the time. We are now not talking about interpretability of the AI, but the reliability; This is easy to identify beforehand. You can also mitigate this (the lightning noise) by including it in the model (data). This should have been glaringly obvious if the person actually knew his toolset rather than just retraining a existing model using tensorflow magic.
  - ssivark 6y ago
    The problem is that a lot of AI folks bury their heads under the sand in the name of “PAC learning” and claim that you’re unfair in asking for out-of-sample generalization. They are loathe to accept that the popular algorithms are overfitting. I’ve had this debate with numerous NN/RL researchers, and somehow it always comes to this impasse.
- mywittyname 6y ago
  I used to go with decision trees where possible for exactly this reason. It's easier to explain than a solution that uses PCA, but has roughly the same results. They are probably not what most people would call ML, but they are simple and solve a number of common problems.
  - mrosett 6y ago
    Single decision trees? Or forests?
    - mywittyname 6y ago
      In most of my work, I could get away with just one big decision tree. I was analyzing user traffic patterns, which has a limited set of metrics associated with it (and lots of boolean values). I wrote a tool that would go through and find "representative examples" of sessions for each tree path, then I could sit down with the UX designers and show them exactly what the user was doing. This was helpful as that group was the most vocal critics of our work, mainly because the data we provided began to drive UX design from senior management, and they weren't happy about it. But when I could sit down and show them exactly how the user was spinning their wheels, they were more likely to buy into changes (and offer effective solutions). Plus, I could inform the UX team which design detail was the primary driver for some event we were testing for. An example would be if the user was applying more than three filters, they were less likely to complete their goal. Thus, the UX team was able to devise a couple of reworks to the filtering mechanisms, which improved customer feedback. This tactic would probably be much more difficult if we used NN to solve the same problem.
  - Der_Einzige 6y ago
    Uhh, PCA is reversable so you can explain the solution of any PCA based transformation nicely, unless you have a different black-box somewhere in your pipeline. If you're in sklearn, the method you want to use is the "inverse transform"
    - nocitrek 6y ago
- WWLink 6y ago
  In other words: Oh dear my fuzzy PID has gone sentient! lol. :)
pwdisswordfish2 6y ago
Can you get your money and time back when it does not perform as promised?
nnq 6y ago
> have any insight to what you can actually deliver Maybe that's the problem. In lots of AI/ML problems you just CAN'T know ahead of time what can be deliver you need to spend the time and resources to do it and then see how well it works... The problem imo is on the business side, most businesses don't know how to transform unpredictable progress into profit (even if average on a large timeframe that progress might be HUGE). So ML/DS people need to overpromise in order to get anything approved, otherwise they'd just have to sit around and do nothing, and overall everyone would be worse too, bc that real but unreliable progress would never happen.
- Jommi 6y ago
  I would love to learn how unpredictability can be leveraged as a positive. Any tips?
  - conjectures 6y ago
    How does insurance work?
    - Jommi 6y ago
      Is this not exactly the opposite? They try to bet that they can predict the occurrence of something better than what you can.
  - nnq 6y ago
    Not pure unpredictability in general, but unpredictability about how much and when progress will be made in a positive direction, eg. what you'd tend to get in applied AI. A few ideas: - in a highly competitive market your actions can be unpredictable by competitors: evolution shows this can work even in the most primitive ways - it's so damn hard to swat a fly because it changes direction randomly, being smarter than it doesn't help... you could get supercharged versions of this advantage with random-AI-progress as long as you don't constrain business direction, eg. if your AI research suddenly produces breakthroughs that shows you can have an advantage on smart adult toys by applying you aerospace technology capabilities, then be willing to get in that space and play to win regardless of how much that makes sense (tip: you'll probably want to stay privately owned, markets will likely hate this "random pivoting" as it would look from outside) -- as business you'd be "the fly that no one can swat" - use what's knows to be good at turning positive unpredictability into overall profits (markets), but internally - have multiple projects & people competing against each other inside a company, cheating antifragility / resilience, and really allow projects to fail (as in make it clear that you could be promoted even if the project/team you lead fails, so encourage the king of risk that would be too high even for a startup) etc. - you'll average out the unpredictibility and be left with the progress + antifragility -- as a business you'll be "the cockroach that survives after radiation, eg. unpredictability, has killed everything else" Hint: all these are already happening, since before AI... and what you'll see now will be that AI/ML will bring most benefits to large corporations with multiple lines of business and multiple independent departments/subsidiaries competing agains each other, and startups-inside-corporations that until now were "pale fakes"... and probably investors already groked this too, there's a lot of growth in AI/ML applied deep inside the guts of big Cos... when this plays out to real wins, result will probably be pretty dystopic: SMBs will be wiped out, lots independent startups too probably (as a small/lone player you can't turn unpredictable progress into profits), you'll get even more centralization, and more mass-surveilance bc now it will not offer only national security advantages but also massive immediate business advantages with AI feeding on the data... EDIT+: That's I think the problem we need to solve is how to de-centralize and spread the benefits of applying AI to business... bc it will naturally promote centralization and closed gardens, unpredictable progress is mostly toxic to each individual player alone... we need some business-version-of-democracy++, pure capitalism will result in a marriage of mega-corps + pseudo-totalitarian-govs since these will be the entities naturally thriving in the new landscape.
astronautjones 6y ago
because it's a grift
- ganstyles 6y ago
  Like many things it can be. And has been used in that way a lot, like other hyped things (blockchain comes to mind). But unlike blockchain, ML has a real use and can be unreasonably effective at certain things. It's not a grift when used for the purposes for which it is effective.
  - EthanHeilman 6y ago
    Blockchain has a real use, in that people use it and it lets people do things they could do before. Blockchain has an expectation problem, its most breathless promoters promised things that no technology can deliver and then when it fails to deliver on those very high expectations it creates the impression that Blockchain failed. Blockchain, for instance, can't fix journalism, but it shouldn't be expected to, anymore than the internet or databases could be expected to "fix journalism".
    - astrophysician 6y ago
      You could say the exact same thing about AI: people who don’t really understand it can dominate a given “AI” culture at a given company, and then others come away with the wrong impression of what it is and what it can and can’t do. To say AI is a grift is a bit silly, but at the same time it’s quite understandable that a lot of people would have this opinion.
paulsutter 6y ago
> It's always the same experience working with AI/ML engineers: This is about as constructive as most political discourse today (disclaimer: I'm working with both robotics and AI/ML folks right now, and they're all terrific)
noelsusman 6y ago
>they had ample time to outline the scope, limitation and requirement before they even started collecting the data. That would be great, but it's just not how data science works. You can't know the scope and limitations of a dataset that doesn't exist yet. In general I agree that data science suffers heavily from the "when all you have is a hammer everything looks like a nail" effect. I see so many projects that attempt to use fancy models when they could be accomplished in an Excel spreadsheet in 15 minutes.
pnw_hazor 6y ago
I have seen a few instances where companies start down the trained models/supervised learning path and then realize it won't work for their use cases. Next, they switch to assisted learning/training using less human labeling, more heuristics, model ensembles (fast learners + slow learners), adversarial models, and so on. Finally they scrap the trained ML classifiers and use other numerical/data science methods. Basically they discover that training effective models takes too much time or too much data or both.

lmilcin 6y ago

I would add "AI" produces results that are hard to understand and work with. When you do "traditional coding" you are getting results that are easier to work with because you understand where they came form. Then it is easier to modify the model and understand the outcome. Your background is robotics. As a robotics example, training AI model to keep bipedal robot upright might be possible but having proper control model that came from control theory, identifying the system, etc. will mean you can actually calculate the envelope of the model, the limitations, etc. and that the model is not likely to surprise you in catastrophic ways. It is a case where AI would be a poor replacement for traditional modelling.

astrophysician 6y ago
I mean, part of the problem is that no one can seem to agree about what "AI" even is anymore; you described a control theory model that "calculate[s] the envelope" which I'm interpreting to mean some sort of confidence interval/posterior, but to me that's exactly what you should be doing whenever possible in an "AI" model, as in, control theory (it sounds like with Gaussian Processes or something) is the right "AI" approach to that problem. To me, linear/logistic regression is AI, as are neural nets, as are pretty much any statistical model. These days "AI" seems to be roughly interpreted as "whatever a DS person deems as a solution to our business problem" at least in my very limited experience. It's all a bit silly since AI has come to mean so many different things that may not fit the traditional definition of AI from e.g. the 90's, but as long as people aren't pidgeon-holeing themselves into particular methods, I'm happy to go along with slapping an "AI" label on pretty much anything.
- sobellian 6y ago
  I took envelope to mean the operating conditions under which the control system is stable. For many control systems this isn't really a confidence interval except insofar as the parameters of the control system are imperfectly known. For as long as I can remember, AI has been a synonym for "hard CS problem" or even just "hard math problem." It has more use as a marketing term than as a technical term with precise meaning.
  - lmilcin 6y ago
    "I took envelope to mean the operating conditions under which the control system is stable." Yes, that's exactly what I meant. I kind of borrowed the term from aviation where this is used to describe conditions under which the behavior of the aircraft is known and predictable. Doesn't mean it is not possible to fly outside the envelope, you just don't get any guarantees.
  - astrophysician 6y ago
    > I took envelope to mean the operating conditions under which the control system is stable. For many control systems this isn't really a confidence interval except insofar as the parameters of the control system are imperfectly known. Ah gotcha, totally agree. > It has more use as a marketing term than as a technical term with precise meaning. yea exactly; not a problem for me that this is the case (buzzwords are just a fact of life), but really what should be happening underneath any "AI" solution is finding and implementing the right solution to the problem. If the right solution is classical control theory or linear models, but the business leaders insist on deep learning because to them that's what AI means, that's where there's a problem.
  - deleted 6y ago
    [deleted]
- numpad0 6y ago
  I’m confident* that “AI” these days means DNN that takes n-dim input in n>2. Anything else is “contemporary“ or “conventional” maths or theories. * Read “not at all”
NalNezumi 6y ago
>I would add "AI" produces results that are hard to understand and work with. Sorry but no, this rhetoric is often used as an easy escape of responsibility but I don't buy it. Sure, if you're talking about the neural nets used in Google translate or YouTube recommendation system, or any big tech company domain of problem that is based on vast amount of data with extreme size of state space and variability - fine. But we are not talking about interpretability of the result but the reliability of it, and in robotics domain you rarely have to tackle that size of state space, but rather a highly noisy one. Let's take your example of low level control of bipedal robot. At whatever stage of control/manipulation you're talking about, you always have to consider that the input will have some noise, and how to handle it. This is the reliability; you know how to(or how much) of the input noise you can handle, and how much you will leave (or amplify) to the rest of the system. Based on this you also know the limit of the system, so you can also get an indication of failure. You can make AI/Neural Network to consider the noise and also give an indication of failure, but you have to think about that from the data gathering step! Most control algorithms have this: "what input noise can I handle" question answered; The same requirement should be set to the data. I assume you thought about Reinforcement Learning of sorts when you thought about control, but what most people don't realize is that those algorithms works only because you can use high precision sensor that reduce these noise level (combined with sensor fusion) to negligible level. But now you're relying on high precision, extremely expensive sensors. They almost always fail when you get out of the lab and have to suddenly consider cost. (and at this stage traditional control algorithm creeps back) Let's take a real example. On my original comment I wrote "work in lab at 1pm" and I weren't sarcastic. Apparently not only did the AI solution (in Computer Vision) I were given only work in the lab, but at the exact lightning condition under high noon at the lab, with all the ceiling lights on all the time. We are now not talking about interpretability of the AI, but the reliability; This is easy to identify beforehand. You can also mitigate this (the lightning noise) by including it in the model (data). This should have been glaringly obvious if the person actually knew his toolset rather than just retraining a existing model using tensorflow magic.
- ssivark 6y ago
  The problem is that a lot of AI folks bury their heads under the sand in the name of “PAC learning” and claim that you’re unfair in asking for out-of-sample generalization. They are loathe to accept that the popular algorithms are overfitting. I’ve had this debate with numerous NN/RL researchers, and somehow it always comes to this impasse.
mywittyname 6y ago
I used to go with decision trees where possible for exactly this reason. It's easier to explain than a solution that uses PCA, but has roughly the same results. They are probably not what most people would call ML, but they are simple and solve a number of common problems.
- mrosett 6y ago
  Single decision trees? Or forests?
  - mywittyname 6y ago
    In most of my work, I could get away with just one big decision tree. I was analyzing user traffic patterns, which has a limited set of metrics associated with it (and lots of boolean values). I wrote a tool that would go through and find "representative examples" of sessions for each tree path, then I could sit down with the UX designers and show them exactly what the user was doing. This was helpful as that group was the most vocal critics of our work, mainly because the data we provided began to drive UX design from senior management, and they weren't happy about it. But when I could sit down and show them exactly how the user was spinning their wheels, they were more likely to buy into changes (and offer effective solutions). Plus, I could inform the UX team which design detail was the primary driver for some event we were testing for. An example would be if the user was applying more than three filters, they were less likely to complete their goal. Thus, the UX team was able to devise a couple of reworks to the filtering mechanisms, which improved customer feedback. This tactic would probably be much more difficult if we used NN to solve the same problem.
- Der_Einzige 6y ago
  Uhh, PCA is reversable so you can explain the solution of any PCA based transformation nicely, unless you have a different black-box somewhere in your pipeline. If you're in sklearn, the method you want to use is the "inverse transform"
  - nocitrek 6y ago
WWLink 6y ago
In other words: Oh dear my fuzzy PID has gone sentient! lol. :)

pwdisswordfish2 6y ago

Can you get your money and time back when it does not perform as promised?

nnq 6y ago

> have any insight to what you can actually deliver Maybe that's the problem. In lots of AI/ML problems you just CAN'T know ahead of time what can be deliver you need to spend the time and resources to do it and then see how well it works... The problem imo is on the business side, most businesses don't know how to transform unpredictable progress into profit (even if average on a large timeframe that progress might be HUGE). So ML/DS people need to overpromise in order to get anything approved, otherwise they'd just have to sit around and do nothing, and overall everyone would be worse too, bc that real but unreliable progress would never happen.

Jommi 6y ago
I would love to learn how unpredictability can be leveraged as a positive. Any tips?
- conjectures 6y ago
  How does insurance work?
  - Jommi 6y ago
    Is this not exactly the opposite? They try to bet that they can predict the occurrence of something better than what you can.
- nnq 6y ago
  Not pure unpredictability in general, but unpredictability about how much and when progress will be made in a positive direction, eg. what you'd tend to get in applied AI. A few ideas: - in a highly competitive market your actions can be unpredictable by competitors: evolution shows this can work even in the most primitive ways - it's so damn hard to swat a fly because it changes direction randomly, being smarter than it doesn't help... you could get supercharged versions of this advantage with random-AI-progress as long as you don't constrain business direction, eg. if your AI research suddenly produces breakthroughs that shows you can have an advantage on smart adult toys by applying you aerospace technology capabilities, then be willing to get in that space and play to win regardless of how much that makes sense (tip: you'll probably want to stay privately owned, markets will likely hate this "random pivoting" as it would look from outside) -- as business you'd be "the fly that no one can swat" - use what's knows to be good at turning positive unpredictability into overall profits (markets), but internally - have multiple projects & people competing against each other inside a company, cheating antifragility / resilience, and really allow projects to fail (as in make it clear that you could be promoted even if the project/team you lead fails, so encourage the king of risk that would be too high even for a startup) etc. - you'll average out the unpredictibility and be left with the progress + antifragility -- as a business you'll be "the cockroach that survives after radiation, eg. unpredictability, has killed everything else" Hint: all these are already happening, since before AI... and what you'll see now will be that AI/ML will bring most benefits to large corporations with multiple lines of business and multiple independent departments/subsidiaries competing agains each other, and startups-inside-corporations that until now were "pale fakes"... and probably investors already groked this too, there's a lot of growth in AI/ML applied deep inside the guts of big Cos... when this plays out to real wins, result will probably be pretty dystopic: SMBs will be wiped out, lots independent startups too probably (as a small/lone player you can't turn unpredictable progress into profits), you'll get even more centralization, and more mass-surveilance bc now it will not offer only national security advantages but also massive immediate business advantages with AI feeding on the data... EDIT+: That's I think the problem we need to solve is how to de-centralize and spread the benefits of applying AI to business... bc it will naturally promote centralization and closed gardens, unpredictable progress is mostly toxic to each individual player alone... we need some business-version-of-democracy++, pure capitalism will result in a marriage of mega-corps + pseudo-totalitarian-govs since these will be the entities naturally thriving in the new landscape.

astronautjones 6y ago

because it's a grift

ganstyles 6y ago
Like many things it can be. And has been used in that way a lot, like other hyped things (blockchain comes to mind). But unlike blockchain, ML has a real use and can be unreasonably effective at certain things. It's not a grift when used for the purposes for which it is effective.
- EthanHeilman 6y ago
  Blockchain has a real use, in that people use it and it lets people do things they could do before. Blockchain has an expectation problem, its most breathless promoters promised things that no technology can deliver and then when it fails to deliver on those very high expectations it creates the impression that Blockchain failed. Blockchain, for instance, can't fix journalism, but it shouldn't be expected to, anymore than the internet or databases could be expected to "fix journalism".
  - astrophysician 6y ago
    You could say the exact same thing about AI: people who don’t really understand it can dominate a given “AI” culture at a given company, and then others come away with the wrong impression of what it is and what it can and can’t do. To say AI is a grift is a bit silly, but at the same time it’s quite understandable that a lot of people would have this opinion.

paulsutter 6y ago

> It's always the same experience working with AI/ML engineers: This is about as constructive as most political discourse today (disclaimer: I'm working with both robotics and AI/ML folks right now, and they're all terrific)

noelsusman 6y ago

>they had ample time to outline the scope, limitation and requirement before they even started collecting the data. That would be great, but it's just not how data science works. You can't know the scope and limitations of a dataset that doesn't exist yet. In general I agree that data science suffers heavily from the "when all you have is a hammer everything looks like a nail" effect. I see so many projects that attempt to use fancy models when they could be accomplished in an Excel spreadsheet in 15 minutes.

pnw_hazor 6y ago

I have seen a few instances where companies start down the trained models/supervised learning path and then realize it won't work for their use cases. Next, they switch to assisted learning/training using less human labeling, more heuristics, model ensembles (fast learners + slow learners), adversarial models, and so on. Finally they scrap the trained ML classifiers and use other numerical/data science methods. Basically they discover that training effective models takes too much time or too much data or both.