7 ms·

How much less care will be taken by the human reviewers knowing that an AI picked the target? There is a bias there in the back of a mind and when you get into

How much less care will be taken by the human reviewers knowing that an AI picked the target? There is a bias there in the back of a mind and when you get into a stressful situation you are more likely to trust that the "system" didn't make a mistake. How many times have you pushed to production skipping the final review because all tests passed anyway only for it to fail?

bandyaboot 3y ago
If I were reviewing AI written code that passes all tests vs reviewing human written code that passes all tests, I think I’d trust the human written code more.
spywaregorilla 3y ago
I knew a guy who was in charge of approving strikes in iraq. His job was assessing the second order impacts of a strike beyond whether or not it was targeting what they believed. I'm not saying iraq wasn't a shit show, but for what it's worth, this stuff is very carefully reviewed and is not done quickly.
- sschueller 3y ago
  Yet the Unites States bombed a Médecins Sans Frontières hospital killing 42 by mistake according to their own non-independent investigation.[1] [1] https://en.wikipedia.org/wiki/Kunduz_hospital_airstrike https://en.wikipedia.org/wiki/Kunduz_hospital_airstrike
  - deleted 3y ago
    [deleted]
  - spywaregorilla 3y ago
    This is tragic but it's noteworthy because it is NOT the norm. Unlike, say, what's going on in Ukraine with frequent missile strikes into arbitrary civilian areas. This is a serious failure of controls that warranted direct attention and the apology of the president. imo this is moreso evidence that the US takes its controls seriously rather than frivolously.
    - ein0p 3y ago
      Both of those things are lies though. Per the British medical journal Lancet, approximately 600000 civilians were killed in Iraq. Per the UN estimates fewer than 11k civilians were killed so far in Ukraine.
      - spywaregorilla 3y ago
        I don't really care enough to get into this, so I won't respond further but that study has largely been debunked. It uses both flawed methodology and appears to have outright fabricated data regarding death certificates. Further, it is grossly misleading to imply that the 600k killed were the result of reckless US military strikes. https://en.wikipedia.org/wiki/Lancet_surveys_of_Iraq_War_casualties https://en.wikipedia.org/wiki/Lancet_surveys_of_Iraq_War_cas...
        ein0p 3y ago
        Wikipedia, really? Ok, let’s set that one aside for the moment. What are your thoughts on this here gem from the first Iraq war: https://m.youtube.com/watch?v=KP1OAD9jSaI https://m.youtube.com/watch?v=KP1OAD9jSaI Is that “humane” or “proportional”?
        spywaregorilla 3y ago
        I think this is a very well documented wikipedia link personally. I'm not interested in following your link.
    - foverzar 3y ago
      > This is tragic but it's noteworthy because it is NOT the norm. Unlike, say, what's going on in Ukraine with frequent missile strikes into arbitrary civilian areas. Let's not dismiss the propaganda factor here. No one caring about their livelihood in the west will ever be digging too deep into US/NATO totally-not-war-crimes-just-collateral-shit-happens-you-know, as well as no one will be arguing Russia's case the same way Ukraine's attacks over Belgorod Region and office buildings in Moscow are excused with "this missile/drone was on its way to a military target, but was shot down over civilians heads by air defense". Assange's case is a very illustrative example what happens when something gets loud enough to actually matter and disrupt the image of righteousness.
- torstenvl 3y ago
  I served as a strike cell attorney in Baghdad during the Daesh fight (Operation Inherent Resolve). Specific criteria are classified, but the U.S. military systematizes the four LOAC principles: military necessity, distinction, proportionality, no unnecessary suffering. Positive identification goes only to the first two principles, and—while I don't know specifics—the DoD's "human in the loop not on the loop" AI philosophy makes me think that AI won't satisfy PID by itself. In addition to that, all munitions targets would have to meet CDE in accordance with CJCSI 3160.01. In my experience, there is typically a CDE expert as well as a lawyer involved in every targeting decision.
  - spywaregorilla 3y ago
    The lawyer bit is fascinating.
  - ein0p 3y ago
    [flagged]
    - sebastos 3y ago
      If you critically examine the source that provided you this number and follow the rabbit hole, you will probably learn that it is wildly pessimistic AND not telling you what you think it’s telling you. E.g. these estimates often include estimated casualties from sectarian violence that was merely uncorked by the Iraq war. You are clearly not taking pains to prevent the casual reader from assuming that 600k (already a specious number) were all directly killed at the hands of US soldiers.
      - overstay8930 3y ago
        The people throwing these kinds of numbers around don't actually care about reality, it's not worth engaging them.
        khokhol 3y ago
        Yep, and they're literally worse than Stalin in this regard. At least he had an appreciation for statistics, and accurate body counts.
- salad-tycoon 3y ago
  Yes as your comment and lawyers comment below show there was a veneer of “we are war professionals, our war is legal and just” and I was listening to every patriotic song that came out celebrating every bomb. My highschool had some highlights of iraq being bombed to the tune of “let the bodies hit the floor.” At the end of the day war is a racket, the prescence of the lawyer detailing what he did below should only raise everyone’s BS meter to “overload.” I’m not sure if it was a million we killed or 600k or maybe less but the defense of “many of those weren’t us, it’s was subsequent totally unrelated non us involved” is also a very lawyerly statement. Like, we only lit this one tree on fire maaaannn, we didn’t light the whole damn neighborhood! Let us not forget the Wikileaks either as a self purported hacker community. 12 unarmed men gleefully murdered by an Apache helicopter (including 2 ap reporters). [1] Let’s also not forget Madeleine Albright defending the death of 1/2million Iraq children in 1996. When asked by 60 Minutes reporter Lesley Stahl "We have heard that half a million children have died. I mean, that's more children than died in Hiroshima. And, you know, is the price worth it?", Albright replied "I think this is a very hard choice, but the price - we think the price is worth it." [2] For what it’s worth, I am a US patriot, I just don’t excuse our BS behavior as I’ve been sold the whole “we are very careful, we have lawyers and 2ndary explosion analysts” line one too many times. The price is always worth it to certain people and industries. I’m not ragging on the lawyer below, I hope he was able to do some good. But it’s all bullshit, from start/middle/end (Afghanistan withdrawal, you should have seen the look on vets faces as they realized every sacrifice, buddy lost, person killed was completely and totally pointless, but don’t worry we have a national suicide call line) 1 https://hackcur.io/collateral-murder-wikileaks-iraq/ https://hackcur.io/collateral-murder-wikileaks-iraq/ 2 https://x.com/theserfstv/status/1506706179178725379?s=46 https://x.com/theserfstv/status/1506706179178725379?s=46
  - spywaregorilla 3y ago
    We are not saying the war was just. It was not a just war. We are saying the US military, as an organization, takes targeting decisions seriously and slowly.
HeyLaughingBoy 3y ago
> How many times have you pushed to production skipping the final review because all tests passed anyway only for it to fail? Never. Because like the gate guard in "Zero Dark Thirty" I understand that process only works if you do it all the time.
pytness 3y ago
Sure, but you wouldnt push to prod on a friday
- bamboozled 3y ago
  I do all the time, what’s your point ?
  - pytness 3y ago
    You dont push on fridays or before holidays. Good luck breaking prod and having you or your coworkers do extra hours on a holiday. edit: the point is you choose when to deploy, eg shoot.
    - mc32 3y ago
      And your customers like it much less if prod breaks during their business hours.
      - pytness 3y ago
        Insurance and banks services are used out of business hours by everyone. You better have people working after pushing to production a friday at 5pm. If it breaks on my work hours, np, i will revert the deployment. Good luck getting me to revert anything on a weekend.
        mc32 3y ago
        Yeah different industries need different approaches. For services which serve North America, you’d preferably have a capable team in Australia, or similar, so you can push changes during their biz hours but our off hours.
chii 3y ago
> How much less care will be taken by the human reviewers knowing that an AI picked the target? it would depend on how the liability flows when the target is wrong. Will the AI cover the ass of the previous person who would've had to vet it? Or will the ultimate responsibility and therefore, liability, lay with a person?
- DrScientist 3y ago
  Personal accountability? This is the US. For example the Captain USS Vincennes got promoted after he ordered the shooting down of a commercial airliner by mistake.
  - singleshot_ 3y ago
    Rogers was a Captain in the navy at the time of the incident. What rank was Captain Rogers promoted to after the incident?
    - DrScientist 3y ago
      You are right I meant the awarding of the Legion of Merit for his time as Captain of the Vincennes.
      - singleshot_ 3y ago
        Oh, wow. I had not known that. Interesting detail for sure!
tromp 3y ago
Use of A.I. should never give you Accountability Immunity...
mdhb 3y ago
There’s really no parallel whatsoever between pushing code to production and a military operation like this. It’s a terrible analogy.
- svrtknst 3y ago
  There are parallells to human complacency, and there have been military incidents due to it.
- acdha 3y ago
  I agree that it’s not a great analogy but would caution that some of the same concerns about complacency apply. If this tool produces some decent looking output, it’s not hard to imagine a mistake being okayed because it looked good similar to someone hitting “LGTM” on Friday afternoon because it’s been fine the last 500 times OR miss something else because they were assuming that the AI would flag the important areas. I know that people in the military are very familiar with problems like that so I think the main concern here is making sure that vendors don’t successfully oversell their products’ capabilities.
  - robertlagrant 3y ago
    It's not hard to imagine, but then the military already uses loads of automation that that applies to. I don't think this increases that risk.
    - acdha 3y ago
      At the very least, it would increase the risk by expanding usage. I think there’s an additional angle, however: the better an automated system like this gets, the more people assume it’ll always work. Tesla drivers get in FSD crashes they’d never get in with a basic cruise control because it’s good enough to handle enough road situations that they assume it can handle all of them, despite periodic fatal reminders that it isn’t.
      - robertlagrant 3y ago
nicklecompte 3y ago
I don't know the technical details behind this software. But I strongly suspect this is firmly in the camp of "machine learning" rather than "AI." Big Tech's intentional confusion between those terms has caused an enormous amount of mischief over the last 10 years. If the military had said "statistical inference using modern Big Data methods" that would have the following advantages: - honesty - officers' expectations would be more tempered and reasonable - public outrage would be more tempered and reasonable Though to be fair this terminology would have the following disadvantages: - military tech grifters have their BS more easily called out - AI tech grifters have to look in the mirror and accept this is what their algorithm is - AI journalism grifters get less clicks - Hacker News conversations indulge less navel-gazing speculation about Skynet
- lesuorac 3y ago
  A thermostat would count as an AI to somebody from the 1800s. My general experience is that AI is just beyond what somebody understands and when they understand how it works they just use a more specific wording.
- whelp_24 3y ago
  Machine Learning is a type of ai. >If the military had said "statistical inference using modern Big Data methods" If this is what I saw the military use I would be upset at the blatant attempt at being misleading. In my mind I would convert it back to AI. It's equivalent to saying "I didn't drive drunk, I got here using carnot-like cycles after injesting social communication enhancing drugs".

bandyaboot 3y ago

If I were reviewing AI written code that passes all tests vs reviewing human written code that passes all tests, I think I’d trust the human written code more.

spywaregorilla 3y ago

I knew a guy who was in charge of approving strikes in iraq. His job was assessing the second order impacts of a strike beyond whether or not it was targeting what they believed. I'm not saying iraq wasn't a shit show, but for what it's worth, this stuff is very carefully reviewed and is not done quickly.

sschueller 3y ago
Yet the Unites States bombed a Médecins Sans Frontières hospital killing 42 by mistake according to their own non-independent investigation.[1] [1] https://en.wikipedia.org/wiki/Kunduz_hospital_airstrike https://en.wikipedia.org/wiki/Kunduz_hospital_airstrike
- deleted 3y ago
  [deleted]
- spywaregorilla 3y ago
  This is tragic but it's noteworthy because it is NOT the norm. Unlike, say, what's going on in Ukraine with frequent missile strikes into arbitrary civilian areas. This is a serious failure of controls that warranted direct attention and the apology of the president. imo this is moreso evidence that the US takes its controls seriously rather than frivolously.
  - ein0p 3y ago
    Both of those things are lies though. Per the British medical journal Lancet, approximately 600000 civilians were killed in Iraq. Per the UN estimates fewer than 11k civilians were killed so far in Ukraine.
    - spywaregorilla 3y ago
      I don't really care enough to get into this, so I won't respond further but that study has largely been debunked. It uses both flawed methodology and appears to have outright fabricated data regarding death certificates. Further, it is grossly misleading to imply that the 600k killed were the result of reckless US military strikes. https://en.wikipedia.org/wiki/Lancet_surveys_of_Iraq_War_casualties https://en.wikipedia.org/wiki/Lancet_surveys_of_Iraq_War_cas...
      - ein0p 3y ago
        Wikipedia, really? Ok, let’s set that one aside for the moment. What are your thoughts on this here gem from the first Iraq war: https://m.youtube.com/watch?v=KP1OAD9jSaI https://m.youtube.com/watch?v=KP1OAD9jSaI Is that “humane” or “proportional”?
        spywaregorilla 3y ago
        I think this is a very well documented wikipedia link personally. I'm not interested in following your link.
  - foverzar 3y ago
    > This is tragic but it's noteworthy because it is NOT the norm. Unlike, say, what's going on in Ukraine with frequent missile strikes into arbitrary civilian areas. Let's not dismiss the propaganda factor here. No one caring about their livelihood in the west will ever be digging too deep into US/NATO totally-not-war-crimes-just-collateral-shit-happens-you-know, as well as no one will be arguing Russia's case the same way Ukraine's attacks over Belgorod Region and office buildings in Moscow are excused with "this missile/drone was on its way to a military target, but was shot down over civilians heads by air defense". Assange's case is a very illustrative example what happens when something gets loud enough to actually matter and disrupt the image of righteousness.
torstenvl 3y ago
I served as a strike cell attorney in Baghdad during the Daesh fight (Operation Inherent Resolve). Specific criteria are classified, but the U.S. military systematizes the four LOAC principles: military necessity, distinction, proportionality, no unnecessary suffering. Positive identification goes only to the first two principles, and—while I don't know specifics—the DoD's "human in the loop not on the loop" AI philosophy makes me think that AI won't satisfy PID by itself. In addition to that, all munitions targets would have to meet CDE in accordance with CJCSI 3160.01. In my experience, there is typically a CDE expert as well as a lawyer involved in every targeting decision.
- spywaregorilla 3y ago
  The lawyer bit is fascinating.
- ein0p 3y ago
  [flagged]
  - sebastos 3y ago
    If you critically examine the source that provided you this number and follow the rabbit hole, you will probably learn that it is wildly pessimistic AND not telling you what you think it’s telling you. E.g. these estimates often include estimated casualties from sectarian violence that was merely uncorked by the Iraq war. You are clearly not taking pains to prevent the casual reader from assuming that 600k (already a specious number) were all directly killed at the hands of US soldiers.
    - overstay8930 3y ago
      The people throwing these kinds of numbers around don't actually care about reality, it's not worth engaging them.
      - khokhol 3y ago
        Yep, and they're literally worse than Stalin in this regard. At least he had an appreciation for statistics, and accurate body counts.
salad-tycoon 3y ago
Yes as your comment and lawyers comment below show there was a veneer of “we are war professionals, our war is legal and just” and I was listening to every patriotic song that came out celebrating every bomb. My highschool had some highlights of iraq being bombed to the tune of “let the bodies hit the floor.” At the end of the day war is a racket, the prescence of the lawyer detailing what he did below should only raise everyone’s BS meter to “overload.” I’m not sure if it was a million we killed or 600k or maybe less but the defense of “many of those weren’t us, it’s was subsequent totally unrelated non us involved” is also a very lawyerly statement. Like, we only lit this one tree on fire maaaannn, we didn’t light the whole damn neighborhood! Let us not forget the Wikileaks either as a self purported hacker community. 12 unarmed men gleefully murdered by an Apache helicopter (including 2 ap reporters). [1] Let’s also not forget Madeleine Albright defending the death of 1/2million Iraq children in 1996. When asked by 60 Minutes reporter Lesley Stahl "We have heard that half a million children have died. I mean, that's more children than died in Hiroshima. And, you know, is the price worth it?", Albright replied "I think this is a very hard choice, but the price - we think the price is worth it." [2] For what it’s worth, I am a US patriot, I just don’t excuse our BS behavior as I’ve been sold the whole “we are very careful, we have lawyers and 2ndary explosion analysts” line one too many times. The price is always worth it to certain people and industries. I’m not ragging on the lawyer below, I hope he was able to do some good. But it’s all bullshit, from start/middle/end (Afghanistan withdrawal, you should have seen the look on vets faces as they realized every sacrifice, buddy lost, person killed was completely and totally pointless, but don’t worry we have a national suicide call line) 1 https://hackcur.io/collateral-murder-wikileaks-iraq/ https://hackcur.io/collateral-murder-wikileaks-iraq/ 2 https://x.com/theserfstv/status/1506706179178725379?s=46 https://x.com/theserfstv/status/1506706179178725379?s=46
- spywaregorilla 3y ago
  We are not saying the war was just. It was not a just war. We are saying the US military, as an organization, takes targeting decisions seriously and slowly.

HeyLaughingBoy 3y ago

> How many times have you pushed to production skipping the final review because all tests passed anyway only for it to fail? Never. Because like the gate guard in "Zero Dark Thirty" I understand that process only works if you do it all the time.

pytness 3y ago

Sure, but you wouldnt push to prod on a friday

bamboozled 3y ago
I do all the time, what’s your point ?
- pytness 3y ago
  You dont push on fridays or before holidays. Good luck breaking prod and having you or your coworkers do extra hours on a holiday. edit: the point is you choose when to deploy, eg shoot.
  - mc32 3y ago
    And your customers like it much less if prod breaks during their business hours.
    - pytness 3y ago
      Insurance and banks services are used out of business hours by everyone. You better have people working after pushing to production a friday at 5pm. If it breaks on my work hours, np, i will revert the deployment. Good luck getting me to revert anything on a weekend.
      - mc32 3y ago
        Yeah different industries need different approaches. For services which serve North America, you’d preferably have a capable team in Australia, or similar, so you can push changes during their biz hours but our off hours.

chii 3y ago

> How much less care will be taken by the human reviewers knowing that an AI picked the target? it would depend on how the liability flows when the target is wrong. Will the AI cover the ass of the previous person who would've had to vet it? Or will the ultimate responsibility and therefore, liability, lay with a person?

DrScientist 3y ago
Personal accountability? This is the US. For example the Captain USS Vincennes got promoted after he ordered the shooting down of a commercial airliner by mistake.
- singleshot_ 3y ago
  Rogers was a Captain in the navy at the time of the incident. What rank was Captain Rogers promoted to after the incident?
  - DrScientist 3y ago
    You are right I meant the awarding of the Legion of Merit for his time as Captain of the Vincennes.
    - singleshot_ 3y ago
      Oh, wow. I had not known that. Interesting detail for sure!

tromp 3y ago

Use of A.I. should never give you Accountability Immunity...

mdhb 3y ago

There’s really no parallel whatsoever between pushing code to production and a military operation like this. It’s a terrible analogy.

svrtknst 3y ago
There are parallells to human complacency, and there have been military incidents due to it.
acdha 3y ago
I agree that it’s not a great analogy but would caution that some of the same concerns about complacency apply. If this tool produces some decent looking output, it’s not hard to imagine a mistake being okayed because it looked good similar to someone hitting “LGTM” on Friday afternoon because it’s been fine the last 500 times OR miss something else because they were assuming that the AI would flag the important areas. I know that people in the military are very familiar with problems like that so I think the main concern here is making sure that vendors don’t successfully oversell their products’ capabilities.
- robertlagrant 3y ago
  It's not hard to imagine, but then the military already uses loads of automation that that applies to. I don't think this increases that risk.
  - acdha 3y ago
    At the very least, it would increase the risk by expanding usage. I think there’s an additional angle, however: the better an automated system like this gets, the more people assume it’ll always work. Tesla drivers get in FSD crashes they’d never get in with a basic cruise control because it’s good enough to handle enough road situations that they assume it can handle all of them, despite periodic fatal reminders that it isn’t.
    - robertlagrant 3y ago

nicklecompte 3y ago

I don't know the technical details behind this software. But I strongly suspect this is firmly in the camp of "machine learning" rather than "AI." Big Tech's intentional confusion between those terms has caused an enormous amount of mischief over the last 10 years. If the military had said "statistical inference using modern Big Data methods" that would have the following advantages: - honesty - officers' expectations would be more tempered and reasonable - public outrage would be more tempered and reasonable Though to be fair this terminology would have the following disadvantages: - military tech grifters have their BS more easily called out - AI tech grifters have to look in the mirror and accept this is what their algorithm is - AI journalism grifters get less clicks - Hacker News conversations indulge less navel-gazing speculation about Skynet

lesuorac 3y ago
A thermostat would count as an AI to somebody from the 1800s. My general experience is that AI is just beyond what somebody understands and when they understand how it works they just use a more specific wording.
whelp_24 3y ago
Machine Learning is a type of ai. >If the military had said "statistical inference using modern Big Data methods" If this is what I saw the military use I would be upset at the blatant attempt at being misleading. In my mind I would convert it back to AI. It's equivalent to saying "I didn't drive drunk, I got here using carnot-like cycles after injesting social communication enhancing drugs".