5 ms·
This sounds like a worryingly bad idea on many levels. 1. "Such a system could find, fix and finish a target without external direction by a human controller."
by ohazi 7y ago
This sounds like a worryingly bad idea on many levels.
1. "Such a system could find, fix and finish a target without external direction by a human controller."
The fact that human-out-of-the-loop kill decisions is a bad idea should not require further elaboration.
2. "Suddenly, a simulated threat radar activated. The pair of networked UCAVs immediately classified the threat and executed a plan to destroy it based on the position of each X-45 in relation to the target, what weapons were available, what each drone’s fuel load was and the nature of the target itself. The calculations happened in a blink of an eye."
Congratulations, now all of the adversarial techniques that make state-of-the-art neural networks fall over will apply to fighter jets as well. Does the system deterministically plan to attack everything it sees that looks like a threat? What about easy threats that show up first, followed by progressively more challenging / fuel-draining threats? Now instead of radar jammers, the signals guys will go back to building elaborate decoy generators.
Without a human in the loop, our current autonomous technologies are woefully inadequate for determining whether or not something smells like a trap. Our enemies are going to learn how to knock these things out of the sky with a $200 SDR and a power amp.
- Zenst 7y agoCan't help but recall a relative episode of StarTrek in which an AI computer takes control of the Enterprise and starts targeting friendlies as it classifies them as hostile. https://en.wikipedia.org/wiki/The_Ultimate_Computer https://en.wikipedia.org/wiki/The_Ultimate_Computer
- outworlder 7y agoThis actually reminds me of an episode of Start Trek: Voyager (Dreadnaught). A missile with an advanced AI gets transported to the Delta quadrant in the same way as Voyager was. It was already en-route to a target planet. Obviously it lost lock. It then used what information it had about the planet to find it again. It found it – except it wasn't the same planet, it was a completely innocent one. And set a course. It correctly calculated that the probability of two planets having the exact same parameters was infinitesimally small, so it must be the correct target. It was further unconvinced that it was, in fact, in the Delta quadrant. It also disregarded the orders to shut down, which came from the same engineer who reprogrammed it, believing that she was a rogue agent. https://memory-alpha.fandom.com/wiki/Dreadnought_(episode) https://memory-alpha.fandom.com/wiki/Dreadnought_(episode)
- Zenst 7y agoAh yes, I remember that one, more you think about it - this lesson has been played out many times in various scifi forms. Even to the extreme of Terminator and skynet. Makes you wonder what will play out in real life as we go into the AI age.
- jandrese 7y agoOr if it has already played out many times in history as people operating on incomplete information have made bad decisions and then stuck with them even in the face of newly incoming conflicting data. Really the worry here is that AIs will be just as bad at decision making as humans. A semi-recent example is the shooting down of MH17, which was shot down by Russians or their allies because they mistook an airliner for a military aircraft, even when their flight profiles were totally different. The controllers were expecting a military aircraft to arrive around that time and didn't stop to consider the alternatives, even when the target didn't behave properly.
- Zenst 7y agoYes the classic 60's saying from computers "Garbage In Garbage Out" still holds true. Though today we can process that garbage faster and react faster - what could go wrong.
- kingkawn 7y agoEverything we’ve imagined we’ll do
- nwallin 7y agoNot sure if the article was edited before you posted, but the author covers this. There is a human in the loop. The swarm sends all its data to the ground station, where a human gives go/no-go. There thing that is automated is the first phase threat identification and the generation and execution of the attack plan, not the threat verification or decision to execute. Which is fine IMO.
- ohazi 7y agoThey say that this is possible, but they're cagey about what they expect the default to be.
- Spooky23 7y agoThat's always in the narrative to make you feel better. If the military gave a shit decisions about attack plans, we wouldn't have missiles. With a plane-sized stealth drone, you can use inertia/gyro guidance to eliminate the GPS risk. Seems silly to build a stealth aircraft that has to transmit and receive instructions. Where do vehicles like this fit? You have long-range cruise missiles, cheap, long loitering drones with hellfire missiles, manned aircraft, and these unmanned fighter-jet class devices already. A weapon like this wouldn't have super-long loitering time, but would be potentially more stealthy and more disposable than a manned aircraft. It's pretty obvious why it's secret... it's an air-defense buster, strike platform against a nation-state (vs. rabble), delivery truck for special bombs, etc or a nuclear weapon delivery truck.
- JackFr 7y ago> It's pretty obvious why it's secret... it's an air-defense buster, strike platform against a nation-state (vs. rabble), delivery truck for special bombs, etc or a nuclear weapon delivery truck. This
- nradov 7y agoInertial guidance isn't accurate enough for that type of mission. The practical alternatives to GPS are terrain matching or stellar navigation.
- fvdessen 7y ago> Without a human in the loop, our current autonomous technologies are woefully inadequate for determining whether or not something smells like a trap. Our enemies are going to learn how to knock these things out of the sky with a $200 SDR and a power amp. If AI can beat humans at strategy & bluffing games like Starcraft & poker, why couldn't they defeat humans at detecting & defeating anti air traps ?
- ohazi 7y agoThose are games with rules and mechanics that are well-specified. A real-world battlefield environment is not nearly as well-specified. Sensors are noisy and can be tricked in all sorts of silly ways. Also the goal isn't necessarily to beat the other player -- making their planning system crash or lose confidence in itself can be more than adequate. I don't think these problems are comparable.
- mlonkibjuyhv 7y agoYou're forgetting it's not actually AI. They'll perform well at whatever task we adequately train them to do. It's an arms race - we train them to defeat anti air traps, the adversary will move the goalpost somewhere else.
- ahupp 7y ago> human-out-of-the-loop kill decisions is a bad idea Maybe so, but this is an inevitable arms race. What happens when the other party has drones and we ned to respond faster than the human can make the decision? > now all of the adversarial techniques that make state-of-the-art neural networks fall over will apply to fighter jets as well. ... Without a human in the loop , our current autonomous technologies are woefully inadequate for determining whether or not something smells like a trap. I'm not sure humans help here. The interesting adversarial examples today are complex images that humans are uniquely good at parsing. But these things are doing to be consuming radar, signals, etc that humans don't really have an advantage with. Not saying decoys etc won't happen, but I don't think humans help here.