7 ms·

But anyone half that smart would know not to give an unproven system absolute, unchecked control over their critical infrastructure, which would basically be a

by Retra 10y ago

But anyone half that smart would know not to give an unproven system absolute, unchecked control over their critical infrastructure, which would basically be a requirement to causing problems humans can't easily overcome.

manicdee 10y ago
The Machines will come to control our infrastructure through a series of small decisions that seemed like a good idea at the time. Computer control of farming decisions such as watering and fertilising. Link that in with insolation and wind prediction to minimise waste or maximise productivity. This happens in the cloud thanks to insolation prediction being tied into wide-area networks of solar farms. More such networks evolve independently, then one day some bright spark realises that prediction and control in domain X can be combined with control and prediction in domain Y to produce greater efficiencies. Linking reproductive health and food production to maximise water savings, for example. Next thing you know the computers decide that human population is best maintained in the order of hundreds of millions rather than billions. And then we find out that Terminator was actually propaganda from the human rebels who didn't like the computers telling them not to breed, and the nuclear war was Skynet protecting its desired humans from the rebels, not protecting itself.
- nradov 10y ago
  The limiting factor in farm productivity is not decision making by human farmers. Anyone who has been farming for years already makes pretty close to optimal decisions based on the available data about crop prices, weather, soil status, labor rates, etc. The ones who make bad decisions end up bankrupt, and leave farming or work as hired farmhands. And there are so few decision-making farmers left anyway that automating the decision making part of their jobs won't achieve any meaningful cost savings.
  - kpil 10y ago
    Maybe Amazon will own 90% of the farms because the have the cheapest robots and best algorithms and statistics.
    - nradov 10y ago
      John Deere and their competitors are far, far ahead of Amazon.com in farm automation. There's a huge difference between warehouse picking robots and farm equipment. Being cheaper doesn't matter, it's ROI that counts.
      - Danihan 10y ago
        Except Amazon could easily acquire John Deere.
        ShannonAlther 10y ago
        Just briefly scanning the numbers on Wikipedia, Amazon.com could acquire John Deere, but it'd be highly unlikely unless they expected that a change in management would yield implausibly high returns.
        Danihan 10y ago
        I suppose my point is, they could acquire the knowledge needed to do cutting edge farm automation if that's something they wanted. And it should be something they want.
        nradov 10y ago
        I'm sure Jeff Bezos is eagerly awaiting your advice on Amazon.com acquisition targets. Farm automation is a great strategic fit with web services and online retailing, right?
        Danihan 10y ago
        It's a rather straightforward step in controlling the supply chain when you are pushing something like Amazon Fresh. I suppose they also aren't a hardware manufacturer? Or a logistics company?
  - alexwebb2 10y ago
    > Anyone who has been farming for years already makes pretty close to optimal decisions based on the available data about crop prices, weather, soil status, labor rates, etc. I'm not sure if you're including software analysis in your notion of "based on the available data", but if not, then this would absolutely not be true - unaided humans, even veteran growers who've been managing the same fields for decades, do not make anywhere close to optimal decisions. They make good ones, generally, and can be long-term profitable with relative ease, but it's absolutely not even close to what you get when you bring intelligent software into the mix.
    - nradov 10y ago
      Sure of course good farmers are taking advantage of the best available software analysis tools. The point is that those tools already work well enough that even if superhuman AI actually becomes available at some point (and that might never happen) it won't necessarily make any better decisions than what human farmers are already doing today.
crusso 10y ago
Would you imagine that a computer smarter than the smartest human and having the understanding of computer software that probably only a computer would have couldn't conceal what it's doing as it creates trojans and viruses that can give it control over every piece of connected equipment everywhere?
- Retra 10y ago
  All of that is moot when you can just turn off the power.
  - lgas 10y ago
    If it was successfully concealing its actions from you, what reason would you have to turn off the power?
    - Retra 10y ago
      If it were successfully concealing its actions, what wrong would it be doing? If all the bad things someone does has no effect on anyone else, then who cares if they do them?
      - crusso 10y ago
        Because it could be concealing its actions until it controls enough infrastructure or military power to launch an attack or prevent people from being able to shut it down. What we don't know can hurt us.
        Retra 10y ago
        It is not 'concealing its actions' if it is taking control of infrastructure and military power. Just like I'm not 'concealing my actions' by putting a blindfold over your eyes. My action is "putting a blindfold over your eyes", which must be very apparent for the entire action to have any value. Now if I hide that action by putting on an invisible blindfold, then you've got nothing to complain about anyway.
MarkPNeyer 10y ago
Just how much faith do you have in your politicians? By using existing political structures, and a mix of rhetoric, they have managed to get a lot more power than a typical person.
- DubiousPusher 10y ago
  Nice point. The US presidential election has had me thinking about how a smart AI might decide instead to manipulate humans into it's will with misinformation campaigns. That of course would require an AI of human comparable intelligence.
  - ben_w 10y ago
    I believe it doesn't necessarily need to outsmart the /average/ human. Consider this scenario: * Someone makes an AI which scores only 85 on IQ tests, but which can run in real time on common consumer hardware. * It gets installed on every phone, PC, and tablet. * Someone, possibly the AI itself if is the type to generate its own goals, sets all instances of the AI the goal of convincing their user of some belief or policy or whatever. * Everyone with IQ<85 is convinced, nobody with an IQ>85 is convinced. * 16% of the population believe whatever it is the goal-setting mind wanted them to believe, at marginal cost. An election is swung, a bank suffers a run, a pyramid scheme makes millions, whatever it is.
    - MarkPNeyer 10y ago
      the ability to emit words on the screen gives a human-like intelligence all kinds of abilities especially if the human-like intelligence doesn't need to pay rent, fear the police, follow laws, sleep, eat ... etc
      - ben_w 10y ago
        In the absence of counter examples, it's reasonable to assume a truly human-like intelligence needs as much down-time and sleep as a real human, and that it can be as effectively coerced with the threat of simulated prison if it breaks the rules. But the point about food and rent is very economically important if they ever get to be cheaper than human labour.
szc 10y ago
This makes me want to find and watch "Colossus: The Forbin Project" (1970s SCI-FI)
ben_w 10y ago
Counter-examples exist every time any critical infrastructure is hacked: if the systems were truly proven, they wouldn't be hackable. Proof is hard, good enough gets deployed.

by Retra 10y ago

manicdee 10y ago

The Machines will come to control our infrastructure through a series of small decisions that seemed like a good idea at the time. Computer control of farming decisions such as watering and fertilising. Link that in with insolation and wind prediction to minimise waste or maximise productivity. This happens in the cloud thanks to insolation prediction being tied into wide-area networks of solar farms. More such networks evolve independently, then one day some bright spark realises that prediction and control in domain X can be combined with control and prediction in domain Y to produce greater efficiencies. Linking reproductive health and food production to maximise water savings, for example. Next thing you know the computers decide that human population is best maintained in the order of hundreds of millions rather than billions. And then we find out that Terminator was actually propaganda from the human rebels who didn't like the computers telling them not to breed, and the nuclear war was Skynet protecting its desired humans from the rebels, not protecting itself.

nradov 10y ago
The limiting factor in farm productivity is not decision making by human farmers. Anyone who has been farming for years already makes pretty close to optimal decisions based on the available data about crop prices, weather, soil status, labor rates, etc. The ones who make bad decisions end up bankrupt, and leave farming or work as hired farmhands. And there are so few decision-making farmers left anyway that automating the decision making part of their jobs won't achieve any meaningful cost savings.
- kpil 10y ago
  Maybe Amazon will own 90% of the farms because the have the cheapest robots and best algorithms and statistics.
  - nradov 10y ago
    John Deere and their competitors are far, far ahead of Amazon.com in farm automation. There's a huge difference between warehouse picking robots and farm equipment. Being cheaper doesn't matter, it's ROI that counts.
    - Danihan 10y ago
      Except Amazon could easily acquire John Deere.
      - ShannonAlther 10y ago
        Just briefly scanning the numbers on Wikipedia, Amazon.com could acquire John Deere, but it'd be highly unlikely unless they expected that a change in management would yield implausibly high returns.
        Danihan 10y ago
        I suppose my point is, they could acquire the knowledge needed to do cutting edge farm automation if that's something they wanted. And it should be something they want.
        nradov 10y ago
        I'm sure Jeff Bezos is eagerly awaiting your advice on Amazon.com acquisition targets. Farm automation is a great strategic fit with web services and online retailing, right?
        Danihan 10y ago
        It's a rather straightforward step in controlling the supply chain when you are pushing something like Amazon Fresh. I suppose they also aren't a hardware manufacturer? Or a logistics company?
- alexwebb2 10y ago
  > Anyone who has been farming for years already makes pretty close to optimal decisions based on the available data about crop prices, weather, soil status, labor rates, etc. I'm not sure if you're including software analysis in your notion of "based on the available data", but if not, then this would absolutely not be true - unaided humans, even veteran growers who've been managing the same fields for decades, do not make anywhere close to optimal decisions. They make good ones, generally, and can be long-term profitable with relative ease, but it's absolutely not even close to what you get when you bring intelligent software into the mix.
  - nradov 10y ago
    Sure of course good farmers are taking advantage of the best available software analysis tools. The point is that those tools already work well enough that even if superhuman AI actually becomes available at some point (and that might never happen) it won't necessarily make any better decisions than what human farmers are already doing today.

crusso 10y ago

Would you imagine that a computer smarter than the smartest human and having the understanding of computer software that probably only a computer would have couldn't conceal what it's doing as it creates trojans and viruses that can give it control over every piece of connected equipment everywhere?

Retra 10y ago
All of that is moot when you can just turn off the power.
- lgas 10y ago
  If it was successfully concealing its actions from you, what reason would you have to turn off the power?
  - Retra 10y ago
    If it were successfully concealing its actions, what wrong would it be doing? If all the bad things someone does has no effect on anyone else, then who cares if they do them?
    - crusso 10y ago
      Because it could be concealing its actions until it controls enough infrastructure or military power to launch an attack or prevent people from being able to shut it down. What we don't know can hurt us.
      - Retra 10y ago
        It is not 'concealing its actions' if it is taking control of infrastructure and military power. Just like I'm not 'concealing my actions' by putting a blindfold over your eyes. My action is "putting a blindfold over your eyes", which must be very apparent for the entire action to have any value. Now if I hide that action by putting on an invisible blindfold, then you've got nothing to complain about anyway.

MarkPNeyer 10y ago

Just how much faith do you have in your politicians? By using existing political structures, and a mix of rhetoric, they have managed to get a lot more power than a typical person.

DubiousPusher 10y ago
Nice point. The US presidential election has had me thinking about how a smart AI might decide instead to manipulate humans into it's will with misinformation campaigns. That of course would require an AI of human comparable intelligence.
- ben_w 10y ago
  I believe it doesn't necessarily need to outsmart the /average/ human. Consider this scenario: * Someone makes an AI which scores only 85 on IQ tests, but which can run in real time on common consumer hardware. * It gets installed on every phone, PC, and tablet. * Someone, possibly the AI itself if is the type to generate its own goals, sets all instances of the AI the goal of convincing their user of some belief or policy or whatever. * Everyone with IQ<85 is convinced, nobody with an IQ>85 is convinced. * 16% of the population believe whatever it is the goal-setting mind wanted them to believe, at marginal cost. An election is swung, a bank suffers a run, a pyramid scheme makes millions, whatever it is.
  - MarkPNeyer 10y ago
    the ability to emit words on the screen gives a human-like intelligence all kinds of abilities especially if the human-like intelligence doesn't need to pay rent, fear the police, follow laws, sleep, eat ... etc
    - ben_w 10y ago
      In the absence of counter examples, it's reasonable to assume a truly human-like intelligence needs as much down-time and sleep as a real human, and that it can be as effectively coerced with the threat of simulated prison if it breaks the rules. But the point about food and rent is very economically important if they ever get to be cheaper than human labour.

szc 10y ago

This makes me want to find and watch "Colossus: The Forbin Project" (1970s SCI-FI)

ben_w 10y ago

Counter-examples exist every time any critical infrastructure is hacked: if the systems were truly proven, they wouldn't be hackable. Proof is hard, good enough gets deployed.