20 ms·

US Marines defeat DARPA robot by hiding under a cardboard box

469 points by koolba 4y ago

exabrial 4y ago
This sounds like a really fun day at work to me.
Too 4y ago
As humiliating this is for the ai. Nobody would have the balls to pull this off in a real battlefield outside of training. Because you never know if you found the perfect camouflage or if you are a sitting duck walking straight into a trap.
8bitsrule 4y ago
Reminds me of a quick exchange in a movie. "Are you human?" "No, I'm a meat popsicle."
bell-cot 4y ago
[flagged]
- burbankio 4y ago
  I like "AI is anything that doesn't work yet".
  - anticensor 4y ago
    Or, absent intelligence.
paradox242 4y ago
I imagined based on the title that they would basically have to include it, and even though I was expecting it, I was still delighted to see a screen cap of Snake with a box over his head. Once the AI has worked it's way through all the twists and turns of the Metal Gear series we are probably back in trouble, though.
jqpabc123 4y ago
This is a good example of the type issues "full self driving" is likely to encounter once it is widely deployed. The real shortcoming of "AI" is that it is almost entirely data driven. There is little to no real cognition or understanding or judgment involved. The human brain can instantly and instinctively extrapolate from what it already knows in order to evaluate and make judgments in new situations it has never seen before. A child can recognize that someone is hiding under a box even if they have never actually seen anyone do it before. Even a dog could likely do the same. AI; as it currently exists, just doesn't do this. It's all replication and repetition. Like any other tool, AI can be useful. But there is no "intelligence" --- it's basically as dumb as a hammer.
- smileysteve 4y ago
  Instantly? Instinctively? Let me introduce you to "peek-a-boo", a simple parent child game for infants. https://en.m.wikipedia.org/wiki/Peekaboo https://en.m.wikipedia.org/wiki/Peekaboo > In early sensorimotor stages, the infant is completely unable to comprehend object permanence.
  - jqpabc123 4y ago
    You do realize there is a difference between an infant and a child, right? An infant will *grow* and develop into a child that is capable of learning and making judgments on it's own. AI never does this. Play "peek-a-boo" with an infant and it will learn and extrapolate from this info and eventually be able to recognize a person hiding under a box even if it has never actually seen it before. AI won't.
    - htrp 4y ago
      >AI never does this. AI never does this now... We're probably one or two generational architecture changes from a system that can do it.
      - jqpabc123 4y ago
        You do realize that people have been making predictions just like yours for decades? "Real" AI is perpetually just around the corner.
prometheus76 4y ago
A hypothetical situation: AI is tied to a camera of me in my office. Doing basic object identification. I stand up. AI recognizes me, recognizes desk. Recognizes "human" and recognizes "desk". I sit on desk. Does AI mark it as a desk or as a chair? And let's zoom in on the chair. AI sees "chair". Slowly zoom in on arm of chair. When does AI switch to "arm of chair"? Now, slowly zoom back out. When does AI switch to "chair"? And should it? When does a part become part of a greater whole, and when does a whole become constituent parts? In other words, we have made great strides in teaching AI "physics" or "recognition", but we have made very little progress in teaching it metaphysics (categories, in this case) because half the people working on the problem don't even recognize metaphysics as a category even though without it, they could not perceive the world. Which is also why AI cannot perceive the world the way we do: no metaphysics.
- feoren 4y ago
  > we have made very little progress in teaching it metaphysics (categories, in this case) That's because ontology, metaphysics, categorization, and all that, is completely worthless bullshit. It's a crutch our limited human brains use, and it causes all sorts of problems. Half of what I do in data modeling is trying to fight against all of the worthless categorizations I come across. There Is No Shelf. Why are categories so bad? Two reasons: 1. They're too easily divorced from their models. Is a tomato a fruit? The questions is faulty: there's no such thing as a "fruit" without a model behind it. When people say "botanically, a tomato is a fruit", they're identifying their model: botany. Okay, are you bio-engineering plants? Or are you cooking dinner? You're cooking dinner. So a tomato is not a fruit. Because when you're cooking dinner, your model is not Botany, it's something culinary, and in any half-decent culinary model, a tomato is a vegetable, not a fruit. So unless we're bio-engineering some plants, shut the hell up about a tomato being a fruit. It's not wisdom/intelligence, it's spouting useless mouth-garbage. And remember that all models are wrong, but some models are useful. Some! Not most. Most models are shit. Categories divorced from a model are worthless, and categories of a shit model are shit. 2. Even good categories of useful models have extremely fuzzy boundaries, and we too often fall into the false dichotomy of thinking something must either "be" or "not be" part of a category. Is an SUV a car? Is a car with a rocket engine on it still a car? Is a car with six wheels still a car? Who cares!? If you're charging tolls for your toll bridge, you instead settle for some countable characteristic like number of axles, and you amend this later if you start seeing lots of vehicles with something that stretches your definition of "axle". In fact the category "car" is worthless most of the time. It's an OK noun, but nouns are only averages; only mental shortcuts to a reasonable approximation of the actual object. If you ever see "class Car : Vehicle", you know you're working in a shit, doomed codebase. And yet you waste time arguing over the definitions of these shit, worthless categories. These worthless things become central to your database and software designs and object hierarchies. Of course you end up with unmaintainable shit. Edit: Three reasons! 3. They're always given too much weight. Male/female: PICK ONE. IT'S VERY IMPORTANT THAT YOU CHOOSE ONE! It is vastly important to our music streaming app that we know whether your skin is black or instead that your ancestors came from the Caucus Mountains or Mongolia. THOSE ARE YOUR ONLY OPTIONS PICK ONE! Employee table: required foreign key to the "Department" table. Departments are virtually meaningless and change all the time! Every time you get a new vice president sitting in some operations chair, the first thing he does is change all the Departments around. You've got people in your Employee table whose department has changed 16 times, but they're the same person, aren't they? Oh, and they're not called "Departments" anymore, they're now "Divisions". Did you change your field name? No, you didn't. Of course you didn't. You have some Contractors in your Employee table, don't you? Some ex-employees that you need to keep around so they show up on that one report? Yeah, you do. Of course you do. Fuck ontology.
aaron695 4y ago
"US Marines Defeat land mine by stepping over it" None of these would work in the field. It's both interesting and pointless. If they didn't work you've increased the robots effectiveness. ie. running slower because you're carrying a fir tree or a box. If the robot has any human backup you are also worse off. Anything to confuse the AI has to not hinder you. A smoke bomb with thermal. It's not clear why the DARPA robot didn't have thermal unless this is a really old story.
- ceejayoz 4y ago
  DARPA isn't doing this with the end goal of advising US troops to bring cardboard boxes along into combat. DARPA is doing this to get AIs that better handle behavior intended to evade AIs.
  - aaron695 4y ago
    [dead]
- roarcher 4y ago
  > It's not clear why the DARPA robot didn't have thermal unless this is a really old story. Who says it didn't? A thermal camera doesn't mean your targets are conveniently highlighted for you and no further identification is needed. Humans aren't the only thing that can be slightly warmer than the background, and on a hot day they may be cooler or blend in. So it's probably best if your robot's target acquisition is a bit more sophisticated than "shoot all the hot things".
insane_dreamer 4y ago
DARPA learning the same lesson the Cylons did: lo-tech saves the day.
euroderf 4y ago
This is where I wonder what the status of Cyc is, and whether it and LLMs can ever live happily together.
smileysteve 4y ago
When you think of this in terms of Western understanding of war, and the perspective that trench warfare was the expectation until post WWII; the conclusions seem incorrect.
DrThunder 4y ago
Hilarious. I immediately heard the Metal Gear exclamation sound in my head when I began reading this.
- kayge 4y ago
  Hah, you beat me to it; Hideo Kojima would be proud. Sounds like DARPA needs to start feeding old stealth video games into their robot's training data :)
  - qikInNdOutReply 4y ago
    But the AI in stealth games is literally trained to go out of its way to not detect you.
    - Firmwarrior 4y ago
      The cardboard box trick doesn't actually work in Metal Gear Solid 2, at least not any better than you'd expect it to work in the real world
      - doubled112 4y ago
        You have to throw a dirty magazine down to distract them first.
        yellow_postit 4y ago
        And have no one question why a produce box is near a nuclear engine/tank/ship/mcguffin.
        doubled112 4y ago
        All it takes is one guard to say "that's how it's always been" and nobody will ever ask questions again.
        4y ago
jeffbee 4y ago
All but literally this technique from BotW https://youtu.be/rAqT9TA-04Y?t=98 https://youtu.be/rAqT9TA-04Y?t=98
DennisP 4y ago
Turns out cats have been preparing for the AI apocalypse all along.
MonkeyMalarky 4y ago
Sounds like they're lacking a second level of interpretation in the system. Image recognition is great. It identifies people, trees and boxes. Object tracking is probably working too, it could follow the people, boxes and trees from one frame to the next. Juuust missing the understanding or belief system that tree+stationary=ok but tree+ambulatory=bad.
- voidfunc 4y ago
  I'd imagine could also look at infrared heat signatures too
  - sethhochberg 4y ago
    Cardboard is a surprisingly effective thermal insulator. But then again, a box that is even slightly warmer than ambient temperature it is... not normal.
    - pazimzadeh 4y ago
      or a box with warm legs sticking out of it? this article reads like a psyops where they want the masses not to be worried
tabtab 4y ago
Soldier A: "Oh no, we're boxed in!" Soldier B: "Relax, it's a good thing."
trabant00 4y ago
I'm surprised they wasted the time and effort to test this instead of just deducing the outcome. Most human jobs that we think we can solve with AI actually require AGI and there is no way around that.
- sovietmudkipz 4y ago
  You kinda need different perspectives and interactions to help build something. E.g. the DARPA engineers thought they had their problem space solved but then some marines did some unexpected stuff. They didn't expect the unexpected, now they can tune their expectations. Seems like the process is working as intended.
closewith 4y ago
Interestingly, the basics of concealment in battle are shape, shine, shadow, silhouette, spacing, surface, and speed (or lack thereof) are all the same techniques the marines used to fool the AI. The boxes and tree changed the silhouette and the somersaults changed the speed of movement. So I guess we've been training soldiers to defeat Skynet all along.
- ridgeguy 4y ago
  Who knew the Marines teach Shakespearean tactics? "Till Birnam wood remove to Dunsinane" Macbeth, Act V, Scene III
  - optimalsolver 4y ago
    That it turned out to just involve regular men with branches stuck to their heads annoyed JRR Tolkien so much that he created the race of Ents.
    - ben_w 4y ago
      I heard the same about caesarean ("none of woman born") becoming a human woman and a male hobbit ("No man can kill me!").
      - PhasmaFelis 4y ago
        I don't know if that's true, but I want it to be true, because the very same thing pissed me off when I read Macbeth in high school.
martin1975 4y ago
Seems we're approaching limits of what is possible w/AI alone. Personally, I find a hybrid approach - interfacing human intelligence w/AI (e.g. like the Borg in ST:TNG?) to provide the military an edge in ways that adversaries cannot easily/quickly reproduce or defeat. There's a reason we still put humans in cockpits even though commercial airliners can pretty much fly themselves.... Hardware and software (AI or anything else) are tools, IMHO, rather than replacements for human beings....
- naasking 4y ago
  > Seems we're approaching limits of what is possible w/AI alone. Not even close. We've barely started in fact.
  - martin1975 4y ago
    How's that? I don't even see problem free self-driving taxis, and they even passed legislation for those in California. There's hype and then there's reality. I get your optimism though.
    - naasking 4y ago
      They've barely started trying. We'd be reaching the limits of AI if self-driving cars were an easy problem and we couldn't quite solve it after 15 years, but self-driving cars are actually a hard problem. Despite that, we're pretty darn close to solving it. There are problems in math that are centuries old, and no one is going around saying we're "reaching the limits of math" just because hard problems are hard.
- pixl97 4y ago
  Humans are hardware we are not anything magical. We do have 4 billion years of evolution keeping our asses alive and that has lead to some very optimized wetware for that effect. But somehow thinking that somehow wetwear is always going to be better than hardware is not a bet I'd make over any 'long' period of time.
PM_me_your_math 4y ago
Devil dogs later discover you can blast DARPA robot into many pieces using the Mk 153.
pugworthy 4y ago
Say you have a convoy of autonomous vehicles traversing a road. They are vision based. You destroy a bridge they will cross, and replace the deck with something like plywood painted to look like a road. They will probably just drive right onto it and fall. Or you put up a "Detour" sign with a false road that leads to a dead end so they all get stuck. As the articles says, "...straight out of Looney Tunes"
- qwerty3344 4y ago
  would humans not make the same mistake?
  - atonse 4y ago
    Maybe. Maybe not. We also have intuition. Where Something just seems fishy. Not saying AI can’t handle that. But I assure you that a human would’ve identified a moving cardboard box as suspicious without being told it’s suspicious. It sounds like this AI was trained more on a whitelist “here are all the possibilities of what marines look like when moving” rather than a black list which is way harder “here are all the things that aren’t suspicious, like what should be an inanimate object changing locations”
    - burnished 4y ago
      Whats special about intuition? Think you could rig up a similar system when your prediction confidence is low.
      - woodson 4y ago
        Part of the problem is that the confidence for “cardboard box” was probably quite high. It’s hard to properly calibrate confidence (speaking from experience, speech recognition is often confidently wrong).
      - atonse 4y ago
        Totally. But it seems like all these ML models are great at image recognition but not behavior recognition. What’s the state of the art with that currently?
grammers 4y ago
Nice story, but we shouldn't trust that technology is not improving further. What we see now is only just the beginning.
- kornhole 4y ago
  The story seems crafted to lull us into not worrying about programmable soldiers and police.
major505 4y ago
The developers didn't played metal gear. The marines did.
AlbertCory 4y ago
They wouldn't defeat a dog that way, though.
antipaul 4y ago
As long as you do something that was _not_ in the training data, you’ll be able to fool the AI robot, right??
eftychis 4y ago
As always Hideo Kojima proves once again to be a visionary.
raydiatian 4y ago
The final word in tactical espionage.
jeffrallen 4y ago
Had an interesting conversation with my 12 year old son about AI tonight. It boiled down to "don't blindly trust ChatGPT, it makes stuff up". Then I encouraged him to try to get it to tell him false/hallucinated things.
amrb 4y ago
A weapon to surpass metal gear!!
barbegal 4y ago
I'm sceptical about this story. It's a nice anecdote for the book to show a point about how training data can't always be generalised to the real world. Unfortunately it just doesn't ring true. Why train it using Marines, don't they have better things to do? And why have the game in the middle of a traffic circle. The whole premise seems just too made up. If anyone has another source corroborating this story (or part of the story) then I'd like to know. But for now I'll assume it's made up to sell the book.
foreverobama 4y ago
[dead]
JoeAltmaier 4y ago
But once an AI is trained to recogniz it, then all the AIs will know. It's the glory of computers - you can load them all with what one has learned.
VLM 4y ago
This is not going to fit well with the groupthink of "ChatGPT and other AI is perfect and going to replace us all"
- brookst 4y ago
  I have literally not seen a single person assert that ChatGPT is perfect. Where are you seeing that? AI will probably, eventually replace most of the tasks we do. That does not mean it replaces us as people, except those who are defined by their tasks.
- mlboss 4y ago
  Anything that requires human body and dexterity is beyond the current state of AI. Anything that is intellectual is within reach. Which makes sense because it took way longer for nature to make human body then it took us to develop language/art/science etc.
- kromem 4y ago
  At this point I've lost track of the number of people who extrapolated from contemporary challenges in AI to predict future shortcomings turning out incredibly wrong within just a few years. It's like there seems to be some sort of bias where over and over when it comes to AI vs human capabilities many humans keep looking at the present and fail to factor in acceleration and not just velocity in their expectations for the future rate of change.
  - ben_w 4y ago
    It's very easy to be wrong as an optimist as well as a pessimist. Back in 2009 I was expecting by 2019 to be able to buy a car in a dealership that didn't have a steering wheel because the self-driving AI would just be that good. Closest we got to that is Waymo taxis in just a few cities. It's good! So is Tesla's thing! Just, both are much much less than I was expecting.
  - varajelle 4y ago
    It's like predicting that flying will never be a mode of transportation while laughing about Wright brothers's planes crashing
Reptur 4y ago
I'm telling you, they're going to have wet towel launchers to defeat these in the future. Or just hold up a poster board in front of you with a mailbox or trash can on it.
kornhole 4y ago
They only need to add thermal engineering to fix this. The terminators are coming John Connor.
coding123 4y ago
https://www.youtube.com/watch?v=BhOc1vyDhjA https://www.youtube.com/watch?v=BhOc1vyDhjA
PhasmaFelis 4y ago
People read stories like this and think "haha, robots are stupid" when they should be thinking "they're identifying the robot's weaknesses so they can fix them."
qwertyuiop_ 4y ago
Marines are trained to improvise, adapt, and overcome all obstacles in all situations. They possess the willingness and the determination to fight and to keep fighting until victory is assured. https://www.marines.com/about-the-marine-corps/who-are-the-marines/purpose.html https://www.marines.com/about-the-marine-corps/who-are-the-m... Looks like the Marines did what they are extremely good at.
andsoitis 4y ago
When intelligence is artificial, understanding and imagination are shallow.
mjevans 4y ago
Wrong training prompt / question. Instead of 'detect a human' the robot should have been trained to detect any unexpected movement or changes in situation.
johnnylambada 4y ago
Oh great, now the robots know the “Dunder Mifflin” maneuver!
hammock 4y ago
This is a good thing. It could mean the autonomous killer robot is less likely to murder someone errantly
kristopolous 4y ago
The takeaway from the "military industrial complex" is instead of optimizing for military acumen, it's a system that optimizes for profit with the sheen of military acumen. So you get weapons that don't work designed for threats that don't exist. It puts the military in a worse place under the illusion that it is in a better place. Not saying anything new here, it's all over 60 years old. Nothing seems to have changed though
- selestify 4y ago
  Events in Ukraine would suggest that even older weapons in the US arsenal do in fact work exceptionally well. US performance in Desert Storm and the battle of Khasam also suggests that the U.S. military does possess the acumen to deploy its weapons effectively.
  - kristopolous 4y ago
    Some niche counterpoint doesn't invalidate the systemic analysis by General Eisenhower. Just like finding a 100 year old smoker doesn't mean smoking doesn't cause cancer. Is this really not immediately obvious?
    - daemoens 4y ago
      Desert Storm was not niche in any shape or form. Iraq was the fourth most powerful military at the time and had the newest and greatest Soviet weapons. The expected casualties for the US were in the thousands, and no one thought the Iraqis would get steamrolled in 1991.
      - kristopolous 4y ago
        Wait wait, hold on Kuwait United States United Kingdom France Saudi Arabia Egypt Afghan mujahideen Argentina Australia Bahrain Bangladesh Belgium Canada Czechoslovakia Denmark Germany Greece Honduras Hungary Italy Japan Morocco Netherlands New Zealand Niger Norway Oman Pakistan Poland Portugal Qatar Senegal Sierra Leone Singapore South Korea Spain Sweden Syria Turkey United Arab Emirates V. Iraq And people placed their bets on Iraq? Using WW2 era T-55s and T-62s from the early 1960s? Alright!
joexner 4y ago
They didn't try very hard to train this system. It wasn't even a prototype. - In the excerpt, Scharre describes a week during which DARPA calibrated its robot’s human recognition algorithm alongside a group of US Marines. The Marines and a team of DARPA engineers spent six days walking around the robot, training it to identify the moving human form. On the seventh day, the engineers placed the robot at the center of a traffic circle and devised a little game: The Marines had to approach the robot from a distance and touch the robot without being detected.
rossjudson 4y ago
Let's keep in mind it's a military robot, so it'll just be shooting boxes on the next iteration. After that it'll be houseplants and suspiciously large cats.
somishere 4y ago
I guess there's more to intelligence than just thinking outside the box ...
chimen 4y ago
The AI is not defeated though, it is being trained.
_448 4y ago
We engineers tend to overlook simple things like these in our grand vision of the world. Yours truly is also at times guilty of getting blinded by these blinders. This reminds me of a joke that was floating around the internet few years ago. It goes something like this: The US and the USSR were in a space race and trying to put a person in space. To give the trainee astronauts a feeling of working without gravity the trainees were trained under water. But the US faced a challenge. There was no pen that would work under water for the trainees to use. The US spent millions of dollars on R&D and finally produced a pen that would work under water. It was a very proud moment for the engineers who developed the technology. After a few months, there was a world conference of space scientists and engineers where teams from both the US and the USSR were also present. So to get a sense of how the USSR team solved the challenge of helping trainees to take notes under water, the US team mention about their own invention and asked the USSR team how they solved the problem. The USSR team replied, We use pencils :)
- louwrentius 4y ago
  The funny thing is that this story has gone round so much yet has been debunked. I can’t remember which podcast it was but in the end you don’t want conducting dust in space and the Russians eventually also bought the space pens. But I still like the point of the story.
davewritescode 4y ago
These types of ML approaches seem to always break down in the face of adversarial input. Security folks are going to have a field day if we’re not careful to make sure that people really understand the limitations of these types of systems.
nix23 4y ago
Next DARPA-robot defeats Marines by being the cardboard box.
perfobotto 4y ago
Snaaaake
jononomo 4y ago
The problem with artificial intelligence is that it is not real intelligence.
gwern 4y ago
I am suspicious of this story, however plausible it seems. The source is given as a book; the Economist writer Shashank Joshi explicitly says that they describe the 'same story' in their article ("(I touched on the same story here: https://economist.com/technology-quarterly/2022/01/27/deception-and-destruction-can-still-blind-the-enemy https://economist.com/technology-quarterly/2022/01/27/decept...)"). However, if you look at the book excerpt (https://twitter.com/shashj/status/1615716082588815363 https://twitter.com/shashj/status/1615716082588815363), it's totally different from the supposed same story (crawl or somersault? cited to Benjamin, or Phil? did the Marine cover his face, or did he cover everything but his face? all undetected, or not the first?). Why should you believe either one after comparing them...? When you have spent much time tracing urban legends, especially in AI where standards for these 'stupid AI stories' are so low that people will happily tell stories with no source ever (https://gwern.net/Tanks https://gwern.net/Tanks) or take an AI deliberately designed to make a particular mistake & erase that context to peddle their error story (eg https://hackernoon.com/dogs-wolves-data-science-and-why-machines-must-learn-like-humans-do-41c43bc7f982 https://hackernoon.com/dogs-wolves-data-science-and-why-mach...), this sort of sloppiness with stories should make you wary.

469 points by koolba 4y ago

This sounds like a really fun day at work to me.

Too 4y ago

As humiliating this is for the ai. Nobody would have the balls to pull this off in a real battlefield outside of training. Because you never know if you found the perfect camouflage or if you are a sitting duck walking straight into a trap.

8bitsrule 4y ago

Reminds me of a quick exchange in a movie. "Are you human?" "No, I'm a meat popsicle."

bell-cot 4y ago

[flagged]

burbankio 4y ago
I like "AI is anything that doesn't work yet".
- anticensor 4y ago
  Or, absent intelligence.

paradox242 4y ago

I imagined based on the title that they would basically have to include it, and even though I was expecting it, I was still delighted to see a screen cap of Snake with a box over his head. Once the AI has worked it's way through all the twists and turns of the Metal Gear series we are probably back in trouble, though.

jqpabc123 4y ago

This is a good example of the type issues "full self driving" is likely to encounter once it is widely deployed. The real shortcoming of "AI" is that it is almost entirely data driven. There is little to no real cognition or understanding or judgment involved. The human brain can instantly and instinctively extrapolate from what it already knows in order to evaluate and make judgments in new situations it has never seen before. A child can recognize that someone is hiding under a box even if they have never actually seen anyone do it before. Even a dog could likely do the same. AI; as it currently exists, just doesn't do this. It's all replication and repetition. Like any other tool, AI can be useful. But there is no "intelligence" --- it's basically as dumb as a hammer.

smileysteve 4y ago
Instantly? Instinctively? Let me introduce you to "peek-a-boo", a simple parent child game for infants. https://en.m.wikipedia.org/wiki/Peekaboo https://en.m.wikipedia.org/wiki/Peekaboo > In early sensorimotor stages, the infant is completely unable to comprehend object permanence.
- jqpabc123 4y ago
  You do realize there is a difference between an infant and a child, right? An infant will *grow* and develop into a child that is capable of learning and making judgments on it's own. AI never does this. Play "peek-a-boo" with an infant and it will learn and extrapolate from this info and eventually be able to recognize a person hiding under a box even if it has never actually seen it before. AI won't.
  - htrp 4y ago
    >AI never does this. AI never does this now... We're probably one or two generational architecture changes from a system that can do it.
    - jqpabc123 4y ago
      You do realize that people have been making predictions just like yours for decades? "Real" AI is perpetually just around the corner.

prometheus76 4y ago

A hypothetical situation: AI is tied to a camera of me in my office. Doing basic object identification. I stand up. AI recognizes me, recognizes desk. Recognizes "human" and recognizes "desk". I sit on desk. Does AI mark it as a desk or as a chair? And let's zoom in on the chair. AI sees "chair". Slowly zoom in on arm of chair. When does AI switch to "arm of chair"? Now, slowly zoom back out. When does AI switch to "chair"? And should it? When does a part become part of a greater whole, and when does a whole become constituent parts? In other words, we have made great strides in teaching AI "physics" or "recognition", but we have made very little progress in teaching it metaphysics (categories, in this case) because half the people working on the problem don't even recognize metaphysics as a category even though without it, they could not perceive the world. Which is also why AI cannot perceive the world the way we do: no metaphysics.

feoren 4y ago
> we have made very little progress in teaching it metaphysics (categories, in this case) That's because ontology, metaphysics, categorization, and all that, is completely worthless bullshit. It's a crutch our limited human brains use, and it causes all sorts of problems. Half of what I do in data modeling is trying to fight against all of the worthless categorizations I come across. There Is No Shelf. Why are categories so bad? Two reasons: 1. They're too easily divorced from their models. Is a tomato a fruit? The questions is faulty: there's no such thing as a "fruit" without a model behind it. When people say "botanically, a tomato is a fruit", they're identifying their model: botany. Okay, are you bio-engineering plants? Or are you cooking dinner? You're cooking dinner. So a tomato is not a fruit. Because when you're cooking dinner, your model is not Botany, it's something culinary, and in any half-decent culinary model, a tomato is a vegetable, not a fruit. So unless we're bio-engineering some plants, shut the hell up about a tomato being a fruit. It's not wisdom/intelligence, it's spouting useless mouth-garbage. And remember that all models are wrong, but some models are useful. Some! Not most. Most models are shit. Categories divorced from a model are worthless, and categories of a shit model are shit. 2. Even good categories of useful models have extremely fuzzy boundaries, and we too often fall into the false dichotomy of thinking something must either "be" or "not be" part of a category. Is an SUV a car? Is a car with a rocket engine on it still a car? Is a car with six wheels still a car? Who cares!? If you're charging tolls for your toll bridge, you instead settle for some countable characteristic like number of axles, and you amend this later if you start seeing lots of vehicles with something that stretches your definition of "axle". In fact the category "car" is worthless most of the time. It's an OK noun, but nouns are only averages; only mental shortcuts to a reasonable approximation of the actual object. If you ever see "class Car : Vehicle", you know you're working in a shit, doomed codebase. And yet you waste time arguing over the definitions of these shit, worthless categories. These worthless things become central to your database and software designs and object hierarchies. Of course you end up with unmaintainable shit. Edit: Three reasons! 3. They're always given too much weight. Male/female: PICK ONE. IT'S VERY IMPORTANT THAT YOU CHOOSE ONE! It is vastly important to our music streaming app that we know whether your skin is black or instead that your ancestors came from the Caucus Mountains or Mongolia. THOSE ARE YOUR ONLY OPTIONS PICK ONE! Employee table: required foreign key to the "Department" table. Departments are virtually meaningless and change all the time! Every time you get a new vice president sitting in some operations chair, the first thing he does is change all the Departments around. You've got people in your Employee table whose department has changed 16 times, but they're the same person, aren't they? Oh, and they're not called "Departments" anymore, they're now "Divisions". Did you change your field name? No, you didn't. Of course you didn't. You have some Contractors in your Employee table, don't you? Some ex-employees that you need to keep around so they show up on that one report? Yeah, you do. Of course you do. Fuck ontology.

aaron695 4y ago

"US Marines Defeat land mine by stepping over it" None of these would work in the field. It's both interesting and pointless. If they didn't work you've increased the robots effectiveness. ie. running slower because you're carrying a fir tree or a box. If the robot has any human backup you are also worse off. Anything to confuse the AI has to not hinder you. A smoke bomb with thermal. It's not clear why the DARPA robot didn't have thermal unless this is a really old story.

ceejayoz 4y ago
DARPA isn't doing this with the end goal of advising US troops to bring cardboard boxes along into combat. DARPA is doing this to get AIs that better handle behavior intended to evade AIs.
- aaron695 4y ago
  [dead]
roarcher 4y ago
> It's not clear why the DARPA robot didn't have thermal unless this is a really old story. Who says it didn't? A thermal camera doesn't mean your targets are conveniently highlighted for you and no further identification is needed. Humans aren't the only thing that can be slightly warmer than the background, and on a hot day they may be cooler or blend in. So it's probably best if your robot's target acquisition is a bit more sophisticated than "shoot all the hot things".

insane_dreamer 4y ago

DARPA learning the same lesson the Cylons did: lo-tech saves the day.

euroderf 4y ago

This is where I wonder what the status of Cyc is, and whether it and LLMs can ever live happily together.

smileysteve 4y ago

When you think of this in terms of Western understanding of war, and the perspective that trench warfare was the expectation until post WWII; the conclusions seem incorrect.

DrThunder 4y ago

Hilarious. I immediately heard the Metal Gear exclamation sound in my head when I began reading this.

kayge 4y ago
Hah, you beat me to it; Hideo Kojima would be proud. Sounds like DARPA needs to start feeding old stealth video games into their robot's training data :)
- qikInNdOutReply 4y ago
  But the AI in stealth games is literally trained to go out of its way to not detect you.
  - Firmwarrior 4y ago
    The cardboard box trick doesn't actually work in Metal Gear Solid 2, at least not any better than you'd expect it to work in the real world
    - doubled112 4y ago
      You have to throw a dirty magazine down to distract them first.
      - yellow_postit 4y ago
        And have no one question why a produce box is near a nuclear engine/tank/ship/mcguffin.
        doubled112 4y ago
        All it takes is one guard to say "that's how it's always been" and nobody will ever ask questions again.
        4y ago

jeffbee 4y ago

All but literally this technique from BotW https://youtu.be/rAqT9TA-04Y?t=98 https://youtu.be/rAqT9TA-04Y?t=98

DennisP 4y ago

Turns out cats have been preparing for the AI apocalypse all along.

MonkeyMalarky 4y ago

Sounds like they're lacking a second level of interpretation in the system. Image recognition is great. It identifies people, trees and boxes. Object tracking is probably working too, it could follow the people, boxes and trees from one frame to the next. Juuust missing the understanding or belief system that tree+stationary=ok but tree+ambulatory=bad.

voidfunc 4y ago
I'd imagine could also look at infrared heat signatures too
- sethhochberg 4y ago
  Cardboard is a surprisingly effective thermal insulator. But then again, a box that is even slightly warmer than ambient temperature it is... not normal.
  - pazimzadeh 4y ago
    or a box with warm legs sticking out of it? this article reads like a psyops where they want the masses not to be worried

tabtab 4y ago

Soldier A: "Oh no, we're boxed in!" Soldier B: "Relax, it's a good thing."

trabant00 4y ago

I'm surprised they wasted the time and effort to test this instead of just deducing the outcome. Most human jobs that we think we can solve with AI actually require AGI and there is no way around that.

sovietmudkipz 4y ago
You kinda need different perspectives and interactions to help build something. E.g. the DARPA engineers thought they had their problem space solved but then some marines did some unexpected stuff. They didn't expect the unexpected, now they can tune their expectations. Seems like the process is working as intended.

closewith 4y ago

Interestingly, the basics of concealment in battle are shape, shine, shadow, silhouette, spacing, surface, and speed (or lack thereof) are all the same techniques the marines used to fool the AI. The boxes and tree changed the silhouette and the somersaults changed the speed of movement. So I guess we've been training soldiers to defeat Skynet all along.

ridgeguy 4y ago
Who knew the Marines teach Shakespearean tactics? "Till Birnam wood remove to Dunsinane" Macbeth, Act V, Scene III
- optimalsolver 4y ago
  That it turned out to just involve regular men with branches stuck to their heads annoyed JRR Tolkien so much that he created the race of Ents.
  - ben_w 4y ago
    I heard the same about caesarean ("none of woman born") becoming a human woman and a male hobbit ("No man can kill me!").
    - PhasmaFelis 4y ago
      I don't know if that's true, but I want it to be true, because the very same thing pissed me off when I read Macbeth in high school.

martin1975 4y ago

Seems we're approaching limits of what is possible w/AI alone. Personally, I find a hybrid approach - interfacing human intelligence w/AI (e.g. like the Borg in ST:TNG?) to provide the military an edge in ways that adversaries cannot easily/quickly reproduce or defeat. There's a reason we still put humans in cockpits even though commercial airliners can pretty much fly themselves.... Hardware and software (AI or anything else) are tools, IMHO, rather than replacements for human beings....

naasking 4y ago
> Seems we're approaching limits of what is possible w/AI alone. Not even close. We've barely started in fact.
- martin1975 4y ago
  How's that? I don't even see problem free self-driving taxis, and they even passed legislation for those in California. There's hype and then there's reality. I get your optimism though.
  - naasking 4y ago
    They've barely started trying. We'd be reaching the limits of AI if self-driving cars were an easy problem and we couldn't quite solve it after 15 years, but self-driving cars are actually a hard problem. Despite that, we're pretty darn close to solving it. There are problems in math that are centuries old, and no one is going around saying we're "reaching the limits of math" just because hard problems are hard.
pixl97 4y ago
Humans are hardware we are not anything magical. We do have 4 billion years of evolution keeping our asses alive and that has lead to some very optimized wetware for that effect. But somehow thinking that somehow wetwear is always going to be better than hardware is not a bet I'd make over any 'long' period of time.

PM_me_your_math 4y ago

Devil dogs later discover you can blast DARPA robot into many pieces using the Mk 153.

pugworthy 4y ago

Say you have a convoy of autonomous vehicles traversing a road. They are vision based. You destroy a bridge they will cross, and replace the deck with something like plywood painted to look like a road. They will probably just drive right onto it and fall. Or you put up a "Detour" sign with a false road that leads to a dead end so they all get stuck. As the articles says, "...straight out of Looney Tunes"

qwerty3344 4y ago
would humans not make the same mistake?
- atonse 4y ago
  Maybe. Maybe not. We also have intuition. Where Something just seems fishy. Not saying AI can’t handle that. But I assure you that a human would’ve identified a moving cardboard box as suspicious without being told it’s suspicious. It sounds like this AI was trained more on a whitelist “here are all the possibilities of what marines look like when moving” rather than a black list which is way harder “here are all the things that aren’t suspicious, like what should be an inanimate object changing locations”
  - burnished 4y ago
    Whats special about intuition? Think you could rig up a similar system when your prediction confidence is low.
    - woodson 4y ago
      Part of the problem is that the confidence for “cardboard box” was probably quite high. It’s hard to properly calibrate confidence (speaking from experience, speech recognition is often confidently wrong).
    - atonse 4y ago
      Totally. But it seems like all these ML models are great at image recognition but not behavior recognition. What’s the state of the art with that currently?

grammers 4y ago

Nice story, but we shouldn't trust that technology is not improving further. What we see now is only just the beginning.

kornhole 4y ago
The story seems crafted to lull us into not worrying about programmable soldiers and police.

major505 4y ago

The developers didn't played metal gear. The marines did.

AlbertCory 4y ago

They wouldn't defeat a dog that way, though.

antipaul 4y ago

As long as you do something that was _not_ in the training data, you’ll be able to fool the AI robot, right??

eftychis 4y ago

As always Hideo Kojima proves once again to be a visionary.

raydiatian 4y ago

The final word in tactical espionage.

jeffrallen 4y ago

Had an interesting conversation with my 12 year old son about AI tonight. It boiled down to "don't blindly trust ChatGPT, it makes stuff up". Then I encouraged him to try to get it to tell him false/hallucinated things.

amrb 4y ago

A weapon to surpass metal gear!!

barbegal 4y ago

I'm sceptical about this story. It's a nice anecdote for the book to show a point about how training data can't always be generalised to the real world. Unfortunately it just doesn't ring true. Why train it using Marines, don't they have better things to do? And why have the game in the middle of a traffic circle. The whole premise seems just too made up. If anyone has another source corroborating this story (or part of the story) then I'd like to know. But for now I'll assume it's made up to sell the book.

foreverobama 4y ago

[dead]

JoeAltmaier 4y ago

But once an AI is trained to recogniz it, then all the AIs will know. It's the glory of computers - you can load them all with what one has learned.

VLM 4y ago

This is not going to fit well with the groupthink of "ChatGPT and other AI is perfect and going to replace us all"

brookst 4y ago
I have literally not seen a single person assert that ChatGPT is perfect. Where are you seeing that? AI will probably, eventually replace most of the tasks we do. That does not mean it replaces us as people, except those who are defined by their tasks.
mlboss 4y ago
Anything that requires human body and dexterity is beyond the current state of AI. Anything that is intellectual is within reach. Which makes sense because it took way longer for nature to make human body then it took us to develop language/art/science etc.
kromem 4y ago
At this point I've lost track of the number of people who extrapolated from contemporary challenges in AI to predict future shortcomings turning out incredibly wrong within just a few years. It's like there seems to be some sort of bias where over and over when it comes to AI vs human capabilities many humans keep looking at the present and fail to factor in acceleration and not just velocity in their expectations for the future rate of change.
- ben_w 4y ago
  It's very easy to be wrong as an optimist as well as a pessimist. Back in 2009 I was expecting by 2019 to be able to buy a car in a dealership that didn't have a steering wheel because the self-driving AI would just be that good. Closest we got to that is Waymo taxis in just a few cities. It's good! So is Tesla's thing! Just, both are much much less than I was expecting.
- varajelle 4y ago
  It's like predicting that flying will never be a mode of transportation while laughing about Wright brothers's planes crashing

Reptur 4y ago

I'm telling you, they're going to have wet towel launchers to defeat these in the future. Or just hold up a poster board in front of you with a mailbox or trash can on it.

kornhole 4y ago

They only need to add thermal engineering to fix this. The terminators are coming John Connor.

coding123 4y ago

https://www.youtube.com/watch?v=BhOc1vyDhjA https://www.youtube.com/watch?v=BhOc1vyDhjA

PhasmaFelis 4y ago

People read stories like this and think "haha, robots are stupid" when they should be thinking "they're identifying the robot's weaknesses so they can fix them."

qwertyuiop_ 4y ago

Marines are trained to improvise, adapt, and overcome all obstacles in all situations. They possess the willingness and the determination to fight and to keep fighting until victory is assured. https://www.marines.com/about-the-marine-corps/who-are-the-marines/purpose.html https://www.marines.com/about-the-marine-corps/who-are-the-m... Looks like the Marines did what they are extremely good at.

andsoitis 4y ago

When intelligence is artificial, understanding and imagination are shallow.

mjevans 4y ago

Wrong training prompt / question. Instead of 'detect a human' the robot should have been trained to detect any unexpected movement or changes in situation.

johnnylambada 4y ago

Oh great, now the robots know the “Dunder Mifflin” maneuver!

hammock 4y ago

This is a good thing. It could mean the autonomous killer robot is less likely to murder someone errantly

kristopolous 4y ago

The takeaway from the "military industrial complex" is instead of optimizing for military acumen, it's a system that optimizes for profit with the sheen of military acumen. So you get weapons that don't work designed for threats that don't exist. It puts the military in a worse place under the illusion that it is in a better place. Not saying anything new here, it's all over 60 years old. Nothing seems to have changed though

selestify 4y ago
Events in Ukraine would suggest that even older weapons in the US arsenal do in fact work exceptionally well. US performance in Desert Storm and the battle of Khasam also suggests that the U.S. military does possess the acumen to deploy its weapons effectively.
- kristopolous 4y ago
  Some niche counterpoint doesn't invalidate the systemic analysis by General Eisenhower. Just like finding a 100 year old smoker doesn't mean smoking doesn't cause cancer. Is this really not immediately obvious?
  - daemoens 4y ago
    Desert Storm was not niche in any shape or form. Iraq was the fourth most powerful military at the time and had the newest and greatest Soviet weapons. The expected casualties for the US were in the thousands, and no one thought the Iraqis would get steamrolled in 1991.
    - kristopolous 4y ago
      Wait wait, hold on Kuwait United States United Kingdom France Saudi Arabia Egypt Afghan mujahideen Argentina Australia Bahrain Bangladesh Belgium Canada Czechoslovakia Denmark Germany Greece Honduras Hungary Italy Japan Morocco Netherlands New Zealand Niger Norway Oman Pakistan Poland Portugal Qatar Senegal Sierra Leone Singapore South Korea Spain Sweden Syria Turkey United Arab Emirates V. Iraq And people placed their bets on Iraq? Using WW2 era T-55s and T-62s from the early 1960s? Alright!

joexner 4y ago

They didn't try very hard to train this system. It wasn't even a prototype. - In the excerpt, Scharre describes a week during which DARPA calibrated its robot’s human recognition algorithm alongside a group of US Marines. The Marines and a team of DARPA engineers spent six days walking around the robot, training it to identify the moving human form. On the seventh day, the engineers placed the robot at the center of a traffic circle and devised a little game: The Marines had to approach the robot from a distance and touch the robot without being detected.

rossjudson 4y ago

Let's keep in mind it's a military robot, so it'll just be shooting boxes on the next iteration. After that it'll be houseplants and suspiciously large cats.

somishere 4y ago

I guess there's more to intelligence than just thinking outside the box ...

chimen 4y ago

The AI is not defeated though, it is being trained.

_448 4y ago

We engineers tend to overlook simple things like these in our grand vision of the world. Yours truly is also at times guilty of getting blinded by these blinders. This reminds me of a joke that was floating around the internet few years ago. It goes something like this: The US and the USSR were in a space race and trying to put a person in space. To give the trainee astronauts a feeling of working without gravity the trainees were trained under water. But the US faced a challenge. There was no pen that would work under water for the trainees to use. The US spent millions of dollars on R&D and finally produced a pen that would work under water. It was a very proud moment for the engineers who developed the technology. After a few months, there was a world conference of space scientists and engineers where teams from both the US and the USSR were also present. So to get a sense of how the USSR team solved the challenge of helping trainees to take notes under water, the US team mention about their own invention and asked the USSR team how they solved the problem. The USSR team replied, We use pencils :)

louwrentius 4y ago
The funny thing is that this story has gone round so much yet has been debunked. I can’t remember which podcast it was but in the end you don’t want conducting dust in space and the Russians eventually also bought the space pens. But I still like the point of the story.

davewritescode 4y ago

These types of ML approaches seem to always break down in the face of adversarial input. Security folks are going to have a field day if we’re not careful to make sure that people really understand the limitations of these types of systems.

nix23 4y ago

Next DARPA-robot defeats Marines by being the cardboard box.

perfobotto 4y ago

Snaaaake

jononomo 4y ago

The problem with artificial intelligence is that it is not real intelligence.

gwern 4y ago

I am suspicious of this story, however plausible it seems. The source is given as a book; the Economist writer Shashank Joshi explicitly says that they describe the 'same story' in their article ("(I touched on the same story here: https://economist.com/technology-quarterly/2022/01/27/deception-and-destruction-can-still-blind-the-enemy https://economist.com/technology-quarterly/2022/01/27/decept...)"). However, if you look at the book excerpt (https://twitter.com/shashj/status/1615716082588815363 https://twitter.com/shashj/status/1615716082588815363), it's totally different from the supposed same story (crawl or somersault? cited to Benjamin, or Phil? did the Marine cover his face, or did he cover everything but his face? all undetected, or not the first?). Why should you believe either one after comparing them...? When you have spent much time tracing urban legends, especially in AI where standards for these 'stupid AI stories' are so low that people will happily tell stories with no source ever (https://gwern.net/Tanks https://gwern.net/Tanks) or take an AI deliberately designed to make a particular mistake & erase that context to peddle their error story (eg https://hackernoon.com/dogs-wolves-data-science-and-why-machines-must-learn-like-humans-do-41c43bc7f982 https://hackernoon.com/dogs-wolves-data-science-and-why-mach...), this sort of sloppiness with stories should make you wary.