5 ms·

It's been my observation that "good enough" ML has been one year out each year for over a decade now. Tesla has a horrible car accident where a car mistakes th

by hajile 5y ago

It's been my observation that "good enough" ML has been one year out each year for over a decade now. Tesla has a horrible car accident where a car mistakes the side of a truck for the sky. They start over and create a new ML system only for the exact same kind of accident to happen again. His points about provable and debuggable making ML radically unsuitable for huge swaths of the tasks where AI would be most desirable are exactly on target. Meanwhile, a tiny little slug seems to understand much more about the universe than the most advanced AI we've ever designed and it does this on nanowatts of power.

mardifoufs 5y ago
My point isn't that ML will end up solving all our problems and lead to AGI. Not at all. What I'm saying is that while it's not perfect, it's still so far beyond anything symbolic AI approaches achieved in the real world that it's just silly to say that we need to "go back" to them. If you think the tesla accidents are bad now, wait until you get a car driven by a hard coded or logic based computer vision system. With ML there is at least a path to solve way-easier-than AGI problems like maybe (semi?) autonomous cars, given enough compute and redundancy. With anything more old-school? Maybe you'd get lane detection that works on sunny days. And yeah you will be able to "debug" and prove things more easily but what's the point if there's no real way to fix the bugs if they are inherent to the flaws of symbolic approaches? If you think Ml has a problem with edge cases (and it does), keep in mind that GOFAI methods are usually not even close to be complete enough to have "edges". "the last 1% is the hardest to deal with" becomes more of a "if we can get over 60% it's a miracle" Also, no matter how suboptimal the tesla autopilot is, I have not seen any figure that show that they do worse than humans w.r.t to fatality or accident rates. Maybe they are worse! But just the fact that they are even remotely close is not something that could've been possible even 15 years ago. But I'd be thrilled if we get to anything better than current DL methods. Hell, it might even pull me back into the field! I just really doubt it's going to be old school symbolic or causal methods that will get us to the next big steps.
- hajile 5y ago
  I believe the answer is somewhere in the middle. Your eyes run on more-or-less fixed-function neural networks to see and recognize stuff. The part of your brain that does the actual driving is much more symbolic in how it analyzes and reacts. When you look at self-driving, there's a general pattern that it handles the 80% of easy stuff (the stuff that you could drive while texting and almost completely ignoring the road). There's a steep gradient down to "complete failure" from there. Accidents happen during a tiny, tiny subset of situations (a minuscule fraction of a percent). Vehicle deaths are in the realm of 1-2 per 100 million miles driven (even lower total accidents if you factor in multiple deaths in one vehicle). If you have a vehicle drive an average of 50mph for 8 hours every day (a very high average if not exclusively driving highways), it would take 700 years for that car to statistically have a fatal accident. For you to test an AI with a decent 95% confidence interval, you'd need hundreds of thousands of cars driving for years just to test a single model. The "enough computation" or "enough data" idea simply isn't going to work here.
  - roenxi 5y ago
    If you need that much evidence to tell that humans are outperforming AI then they may as well be driving to the same standard. Arguing there is a difference when someone might not be able to detect the difference in a lifetime is getting rather lost in the weeds. By the time humans have been proven superior there is a real chance that the AIs will have improved to be superhuman. The experience in gaming is that state of the art goes from ok vs amateurs to superhuman quite quickly.
    - hajile 5y ago
      Humans ARE driving to that standard right now. Deaths per mile are vanishingly small in the grand scheme of things.
      - hmlongco 5y ago
        Yeah, but we shouldn't have a double standard. If we could get an AI to reduce the number of deaths in half it would still be worthwhile. Besides, non-fatal ACCIDENTS per mile are much more common, and we should be able to determine if those are reducing a lot faster.
jjcon 5y ago
>It's been my observation that "good enough" ML has been one year out each year for over a decade now. For self driving cars specifically sure - but ML has its deployments across huge swaths of industry from communications equipment to consumer software today at this moment. To most people these are entirely transparent, but are providing better experiences in tons of products.
- cgio 5y ago
  On the other hand, I’ve seen things such as e.g. approaches which amount to caching, being put forward as ML for database performance optimisation. So there’s also quite a bit of in your face ML that is not even ML to counter the transparent real ML.
  - jjcon 5y ago
    Yeah often the more marketing dollars spent to hype something as AI/ML the less likely it is to be AI/ML - the real applications just speak for themselves in their own utility label or no.
- erosenbe0 5y ago
  Yes, too much focus on AGI. We don't need AGI or self-driving to make a big leap. I want drone fleets that can roof a house or 3d print structures. Nanobots to clear arteries and destroy cancer cells. These things will not need to pass the Turing test or even close.
  - antupis 5y ago
    I just want cheap car where I can space out at wheel and it beeps when I have to continue driving.
    - mellavora 5y ago
      In Europe they call these things a "bus" or a "train". They are usually cheap, in Luxembourg all public transport is free. You get into it, and it takes you to the destination. You can space out all you like. Yes, it is slower than driving yourself, unless there is heavy traffic, but it tends to work well.
IshKebab 5y ago
> one year out each year for over a decade I guess you haven't used Alexa/Ok Google or machine translation or semantic photo search or Google search for several years then?
matt123456789 5y ago
If you’re throwing down the gauntlet, then formalize the description and measurement criteria of slug-related tasks and watch in awe as AlphaSlug slaughters bio slugs in competitive slugfare.
galaxyLogic 5y ago
But it's obvious. Slugs and their ancestors have hit the truck many times and thus learned to avoid them. Tesla just has to train their cars more, but not on public roads. Personally I think we would be better off starting with self-driving trains, there's still a lot of train accidents even with human operators.
- user-the-name 5y ago

by hajile 5y ago

mardifoufs 5y ago

My point isn't that ML will end up solving all our problems and lead to AGI. Not at all. What I'm saying is that while it's not perfect, it's still so far beyond anything symbolic AI approaches achieved in the real world that it's just silly to say that we need to "go back" to them. If you think the tesla accidents are bad now, wait until you get a car driven by a hard coded or logic based computer vision system. With ML there is at least a path to solve way-easier-than AGI problems like maybe (semi?) autonomous cars, given enough compute and redundancy. With anything more old-school? Maybe you'd get lane detection that works on sunny days. And yeah you will be able to "debug" and prove things more easily but what's the point if there's no real way to fix the bugs if they are inherent to the flaws of symbolic approaches? If you think Ml has a problem with edge cases (and it does), keep in mind that GOFAI methods are usually not even close to be complete enough to have "edges". "the last 1% is the hardest to deal with" becomes more of a "if we can get over 60% it's a miracle" Also, no matter how suboptimal the tesla autopilot is, I have not seen any figure that show that they do worse than humans w.r.t to fatality or accident rates. Maybe they are worse! But just the fact that they are even remotely close is not something that could've been possible even 15 years ago. But I'd be thrilled if we get to anything better than current DL methods. Hell, it might even pull me back into the field! I just really doubt it's going to be old school symbolic or causal methods that will get us to the next big steps.

hajile 5y ago
I believe the answer is somewhere in the middle. Your eyes run on more-or-less fixed-function neural networks to see and recognize stuff. The part of your brain that does the actual driving is much more symbolic in how it analyzes and reacts. When you look at self-driving, there's a general pattern that it handles the 80% of easy stuff (the stuff that you could drive while texting and almost completely ignoring the road). There's a steep gradient down to "complete failure" from there. Accidents happen during a tiny, tiny subset of situations (a minuscule fraction of a percent). Vehicle deaths are in the realm of 1-2 per 100 million miles driven (even lower total accidents if you factor in multiple deaths in one vehicle). If you have a vehicle drive an average of 50mph for 8 hours every day (a very high average if not exclusively driving highways), it would take 700 years for that car to statistically have a fatal accident. For you to test an AI with a decent 95% confidence interval, you'd need hundreds of thousands of cars driving for years just to test a single model. The "enough computation" or "enough data" idea simply isn't going to work here.
- roenxi 5y ago
  If you need that much evidence to tell that humans are outperforming AI then they may as well be driving to the same standard. Arguing there is a difference when someone might not be able to detect the difference in a lifetime is getting rather lost in the weeds. By the time humans have been proven superior there is a real chance that the AIs will have improved to be superhuman. The experience in gaming is that state of the art goes from ok vs amateurs to superhuman quite quickly.
  - hajile 5y ago
    Humans ARE driving to that standard right now. Deaths per mile are vanishingly small in the grand scheme of things.
    - hmlongco 5y ago
      Yeah, but we shouldn't have a double standard. If we could get an AI to reduce the number of deaths in half it would still be worthwhile. Besides, non-fatal ACCIDENTS per mile are much more common, and we should be able to determine if those are reducing a lot faster.

jjcon 5y ago

>It's been my observation that "good enough" ML has been one year out each year for over a decade now. For self driving cars specifically sure - but ML has its deployments across huge swaths of industry from communications equipment to consumer software today at this moment. To most people these are entirely transparent, but are providing better experiences in tons of products.

cgio 5y ago
On the other hand, I’ve seen things such as e.g. approaches which amount to caching, being put forward as ML for database performance optimisation. So there’s also quite a bit of in your face ML that is not even ML to counter the transparent real ML.
- jjcon 5y ago
  Yeah often the more marketing dollars spent to hype something as AI/ML the less likely it is to be AI/ML - the real applications just speak for themselves in their own utility label or no.
erosenbe0 5y ago
Yes, too much focus on AGI. We don't need AGI or self-driving to make a big leap. I want drone fleets that can roof a house or 3d print structures. Nanobots to clear arteries and destroy cancer cells. These things will not need to pass the Turing test or even close.
- antupis 5y ago
  I just want cheap car where I can space out at wheel and it beeps when I have to continue driving.
  - mellavora 5y ago
    In Europe they call these things a "bus" or a "train". They are usually cheap, in Luxembourg all public transport is free. You get into it, and it takes you to the destination. You can space out all you like. Yes, it is slower than driving yourself, unless there is heavy traffic, but it tends to work well.

IshKebab 5y ago

> one year out each year for over a decade I guess you haven't used Alexa/Ok Google or machine translation or semantic photo search or Google search for several years then?

matt123456789 5y ago

If you’re throwing down the gauntlet, then formalize the description and measurement criteria of slug-related tasks and watch in awe as AlphaSlug slaughters bio slugs in competitive slugfare.

galaxyLogic 5y ago

But it's obvious. Slugs and their ancestors have hit the truck many times and thus learned to avoid them. Tesla just has to train their cars more, but not on public roads. Personally I think we would be better off starting with self-driving trains, there's still a lot of train accidents even with human operators.

user-the-name 5y ago