11 ms·

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Ne

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

Arn_Thor 2y ago
If any regulator acts it will be the EU. The action, if it comes, will of course be very late, possibly years from now, when the horse has long left the stable.
- sschueller 2y ago
  My only hope for the EU government is that they put and AI in charge and it accidentally becomes sentient...
rolandog 2y ago
> or make surgery decisions? Analyzing surgical field... Identified: open chest cavity, exposed internal organs Organs appear gooey, gelatinous, translucent pink Comparing to database of aquatic lifeforms... 93% visual match found: Psychrolutes marcidus, common name "blobfish" Conclusion: Blobfish discovered inhabiting patient's thoracic cavity Recommended action: Attempt to safely extract blobfish without damaging organs
- varelse 2y ago
  [dead]
hehdhdjehehegwv 2y ago
Right now insurance companies make those decisions based on how your life affects the profit/loss statement at the end of the quarter. (In the USA). So it can’t really be worse if there’s just a RNG in a box. It may be better.
histories 2y ago
> OpenAI is selling this a a tutor for your kids. The Diamond Age.
- ipsin 2y ago
  That's what I find most offensive about the use of LLMs in education: it can readily produce something in the shape of a logical argument, without actually being correct. I'm worried that a generation might learn that that's good enough.
  - Kostchei 2y ago
    a generation of consultants is already doing that- look at the ruckus around PWC etc in Australia. Hell, look at the folks supposedly doing diligence on Enron. This is not new. People lie, fib and prevaricate. The fact the machines trained on our actions do the same thing should not come as a shock. If anything it strikes me as the uncanny valley of truthiness.
- varelse 2y ago
  [dead]
GuardianCaveman 2y ago
I was in a counter-intelligence unit briefly and there was a mathemtician who spoke to us about the work they were doing to pick targets with the idea that if you can only out one person, who would be the most disruptive. You have all these interconnected but mostly isolated terrorist cells that don't know about each other except through a few people who may not be high up in the command but who are critical for the continuing cohesive existence of that terrorist group of cells and logistics etc. So the military already was using math to pick targets, this is just the next logical step, albeit, scary as hell step.
- jspank 2y ago
  In your scenario there were still individuals accountable for the decisions and their outcomes. How are you supposed to say why a machine learning model produces different outputs from the same input? It's just a black box.
kromokromo 2y ago
This is just doomerism. Even though this model is slightly better than the previous, using an LLM for high risk tasks like healthcare and picking targets in military operations still feels very far away. I work in healthcare tech in a European country and yes we use AI for image recognition on x-rays, retinas etc but these are fundamentally completely different models than a LLM. Using LLMs for picking military targets is just absurd. In the future, someone might use some other variation of AI for this but LLMs are not very effective on this.
- ExoticPearTree 2y ago
  I use ChatGPT in particular to narrow down options when I do research, and it is very good at this. It wouldn't be far-fetched to feed it a map, traffic patterns and ask it to do some analysis of "what is the most likeliest place to hit"? And then take it from there.
- dbspin 2y ago
  AI is already being used for picking targets in warzones - https://theconversation.com/israel-accused-of-using-ai-to-target-thousands-in-gaza-as-killer-algorithms-outpace-international-law-227453#:~:text=The%20Israeli%20army%20used%20a,by%20Israeli%20and%20Palestinian%20journalists https://theconversation.com/israel-accused-of-using-ai-to-ta.... LLM's will of course also be used, due to their convenience and superficial 'intelligence', and because of the layer of deniability creating a technical substrate between soldier and civilian victim provides - as has happened for two decades with drones.
  - mike_hearn 2y ago
    Note that the IDF explicitly denied that story: https://www.idf.il/en/mini-sites/hamas-israel-war-24/all-articles/idf-response-as-sent-to-the-guardian/ https://www.idf.il/en/mini-sites/hamas-israel-war-24/all-art... Probably this is due to confusion over what the term "AI" means. If you do some queries on a database, and call yourself a "data scientist", and other people who call themselves data scientists do some AI, does that mean you're doing AI? For left wing journalists who want to undermine the Israelis (the story originally appeared in the Guardian) it'd be easy to hear what you want to hear from your sources and conflate using data with using AI. This is the kind of blurring that happens all the time with apparently technical terms once they leave the tech world and especially once they enter journalism.
fragmede 2y ago
OpenAI is. Their TOS says don't use it for that kind of shit. https://openai.com/policies/usage-policies/ https://openai.com/policies/usage-policies/
- tsimionescu 2y ago
  That's the license for the public service. Nothing prevents them from selling it as a separate package deal to an army.
ComplexSystems 2y ago
Why would the military use ChatGPT or depend on any way on Openai 's policy? Wouldn't they just roll their own?
antihero 2y ago
It is being used to pick military targets, with very little oversight. https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/
nvarsj 2y ago
I’ve had coworkers suggest a technical solution that was straight up fabricated by an LLM and made no sense. More competent people realise this limitation of the models and can use them wisely. Unfortunately I expect to see the former spread.
- meindnoch 2y ago
  I've spent a few hours last week crafting a piece of code for my coworker, and then when I asked him to test it in the real environment, it turned out that the API he wanted to connect to the code I gave him was just a hallucination by ChatGPT.
- denvrede 2y ago
  We had a manager joining last year which, on their first days, created MRs for existing code bases, wrote documents on new processes and gave advice on current problems we were facing. Everything was created by LLMs and plain bullshit. Fortunately were able to convince the higher ups that this person was an imposter and we got rid of them. I really hope that these type of situations won't increase because the mental strain that put on some people in the org is not sustainable in the long run.
booleandilemma 2y ago
People aren't dumb. They'll catch on pretty quick that this thing is BS'ing them.
ethbr1 2y ago
I get a good chuckle every morning when the "C3.ai" ad rolls on NPR. "Hallucination-free," indeed. Would love to know what actual, contractual guarantees they place around that.
farmdve 2y ago
Next it's going to teach them the Earth is flat and there are aliens behind the moon.
CWuestefeld 2y ago
I hear these complaints and can't see how this is worse than the pre-AI situation. How is an AI "hallucination" different from human-generated works that are just plain wrong, or otherwise misleading? Humans make mistakes all the time. Teachers certainly did back when I was in school. There's no fundamental qualitative difference here. And I don't even see any evidence that there's any difference in degree, either.
- lnxg33k1 2y ago
  Probably the main difference is that humans fail at smaller scale, with smaller effects, and build a reputation, probably chatgpt hallucinations can potentially affect everyone
- unclebucknasty 2y ago
  >There's no fundamental qualitative difference here...degree either. I've heard the same comparisons made with self-driving cars (i.e. that humans are fallible, and maybe even more error-prone). But this misses the point. People trust the fallibility they know. That is, we largely understand human failure modes (errors in judgement, lapses in attention, etc) and feel like we are in control of them (and we are). OTOH, when machines make mistakes, they are experienced as unpredictable and outside of our control. Additionally, our expectation of machines is that they are deterministic and not subject to mistakes. While we know bugs can exist, it's not the expectation. And, with the current generation of AI in particular, we are dealing with models that are generally probabilistic, which means there's not even the expectation that they are errorless. And, I don't believe it's reasonable to expect people to give up control to AI of this quality, particularly in matters of safety or life and death; really anything that matters. TLDR; Most people don't want to gamble their lives on a statistic, when the alternative is maintaining control.
  - chaorace 2y ago
    Expanding on this, human failures and machine failures are qualitatively different in ways that make our systems generally less resilient against the machine variety, even when dealing with a theoretically near-perfect implementation. Consider a bug in an otherwise perfect self-driving car routine that causes crashes under a highly specific scenario -- roads are essentially static structures, so you've effectively concentrated 100% of crashes into (for example) 1% of corridors. Practically speaking, those corridors would be forced into a state of perpetual closure. This is all to say that randomly distributed failures are more tolerable than a relatively smaller number of concentrated failures. Human errors are rather nice by comparison because they're inconsistent in locality while still being otherwise predictable in macroscopic terms (e.g.: on any given day, there will always be far more rear-endings than head-on collisions). When it comes to machine networks, all it takes is one firmware update for both the type & locality of their failure modes to go into a wildly different direction.
Dumblydorr 2y ago
Surgeons don’t need a text based LLM to make decisions. They have a job to do and a dozen years of training into how to do it. They have 8 years of schooling and 4-6 years internship and residency. The tech fantasy that everyone is using these for everything is a bubble thought. I agree with another comment, this is Doomerism.
- CuriouslyC 2y ago
  Surgeons are using robots that are far beyond fly by wire though, to the point that you could argue they're instructing the robots rather than controlling them.
HarHarVeryFunny 2y ago
Israel is already doing exactly that... using AI to identify potential targets based on their network of connections, giving these potential targets a cursory human screening, then OK-ing the bombing of their entire family since they have put such faith (and/or just don't care) in this identification process that these are considered high-value targets where "collateral damage" is accepted.
- deleted 2y ago
  [deleted]
bobosha 2y ago
https://pessimistsarchive.org/ https://pessimistsarchive.org/
DeathArrow 2y ago
>Who is going to prevent this from being used to pick military targets When AI is in charge of controlling weapons, you get this: https://www.accessnow.org/publication/artificial-genocidal-intelligence-israel-gaza/ https://www.accessnow.org/publication/artificial-genocidal-i...
chazeon 2y ago
It seems US and China are trying to reach an agreement to use AI to pick military targets these days.
gdubs 2y ago
While this is clearly a problem and a challenge to address, the thing that never gets mentioned with this line of criticism is the obvious: a large number of real-life teachers make mistakes ALL the time. They harbor wrong / out-dated opinions, or they're just flat-out wrong about things.

Arn_Thor 2y ago

If any regulator acts it will be the EU. The action, if it comes, will of course be very late, possibly years from now, when the horse has long left the stable.

sschueller 2y ago
My only hope for the EU government is that they put and AI in charge and it accidentally becomes sentient...

rolandog 2y ago

> or make surgery decisions? Analyzing surgical field... Identified: open chest cavity, exposed internal organs Organs appear gooey, gelatinous, translucent pink Comparing to database of aquatic lifeforms... 93% visual match found: Psychrolutes marcidus, common name "blobfish" Conclusion: Blobfish discovered inhabiting patient's thoracic cavity Recommended action: Attempt to safely extract blobfish without damaging organs

varelse 2y ago
[dead]

hehdhdjehehegwv 2y ago

Right now insurance companies make those decisions based on how your life affects the profit/loss statement at the end of the quarter. (In the USA). So it can’t really be worse if there’s just a RNG in a box. It may be better.

histories 2y ago

> OpenAI is selling this a a tutor for your kids. The Diamond Age.

ipsin 2y ago
That's what I find most offensive about the use of LLMs in education: it can readily produce something in the shape of a logical argument, without actually being correct. I'm worried that a generation might learn that that's good enough.
- Kostchei 2y ago
  a generation of consultants is already doing that- look at the ruckus around PWC etc in Australia. Hell, look at the folks supposedly doing diligence on Enron. This is not new. People lie, fib and prevaricate. The fact the machines trained on our actions do the same thing should not come as a shock. If anything it strikes me as the uncanny valley of truthiness.
varelse 2y ago
[dead]

GuardianCaveman 2y ago

I was in a counter-intelligence unit briefly and there was a mathemtician who spoke to us about the work they were doing to pick targets with the idea that if you can only out one person, who would be the most disruptive. You have all these interconnected but mostly isolated terrorist cells that don't know about each other except through a few people who may not be high up in the command but who are critical for the continuing cohesive existence of that terrorist group of cells and logistics etc. So the military already was using math to pick targets, this is just the next logical step, albeit, scary as hell step.

jspank 2y ago
In your scenario there were still individuals accountable for the decisions and their outcomes. How are you supposed to say why a machine learning model produces different outputs from the same input? It's just a black box.

kromokromo 2y ago

This is just doomerism. Even though this model is slightly better than the previous, using an LLM for high risk tasks like healthcare and picking targets in military operations still feels very far away. I work in healthcare tech in a European country and yes we use AI for image recognition on x-rays, retinas etc but these are fundamentally completely different models than a LLM. Using LLMs for picking military targets is just absurd. In the future, someone might use some other variation of AI for this but LLMs are not very effective on this.

ExoticPearTree 2y ago
I use ChatGPT in particular to narrow down options when I do research, and it is very good at this. It wouldn't be far-fetched to feed it a map, traffic patterns and ask it to do some analysis of "what is the most likeliest place to hit"? And then take it from there.
dbspin 2y ago
AI is already being used for picking targets in warzones - https://theconversation.com/israel-accused-of-using-ai-to-target-thousands-in-gaza-as-killer-algorithms-outpace-international-law-227453#:~:text=The%20Israeli%20army%20used%20a,by%20Israeli%20and%20Palestinian%20journalists https://theconversation.com/israel-accused-of-using-ai-to-ta.... LLM's will of course also be used, due to their convenience and superficial 'intelligence', and because of the layer of deniability creating a technical substrate between soldier and civilian victim provides - as has happened for two decades with drones.
- mike_hearn 2y ago
  Note that the IDF explicitly denied that story: https://www.idf.il/en/mini-sites/hamas-israel-war-24/all-articles/idf-response-as-sent-to-the-guardian/ https://www.idf.il/en/mini-sites/hamas-israel-war-24/all-art... Probably this is due to confusion over what the term "AI" means. If you do some queries on a database, and call yourself a "data scientist", and other people who call themselves data scientists do some AI, does that mean you're doing AI? For left wing journalists who want to undermine the Israelis (the story originally appeared in the Guardian) it'd be easy to hear what you want to hear from your sources and conflate using data with using AI. This is the kind of blurring that happens all the time with apparently technical terms once they leave the tech world and especially once they enter journalism.

fragmede 2y ago

OpenAI is. Their TOS says don't use it for that kind of shit. https://openai.com/policies/usage-policies/ https://openai.com/policies/usage-policies/

tsimionescu 2y ago
That's the license for the public service. Nothing prevents them from selling it as a separate package deal to an army.

ComplexSystems 2y ago

Why would the military use ChatGPT or depend on any way on Openai 's policy? Wouldn't they just roll their own?

antihero 2y ago

It is being used to pick military targets, with very little oversight. https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/

nvarsj 2y ago

I’ve had coworkers suggest a technical solution that was straight up fabricated by an LLM and made no sense. More competent people realise this limitation of the models and can use them wisely. Unfortunately I expect to see the former spread.

meindnoch 2y ago
I've spent a few hours last week crafting a piece of code for my coworker, and then when I asked him to test it in the real environment, it turned out that the API he wanted to connect to the code I gave him was just a hallucination by ChatGPT.
denvrede 2y ago
We had a manager joining last year which, on their first days, created MRs for existing code bases, wrote documents on new processes and gave advice on current problems we were facing. Everything was created by LLMs and plain bullshit. Fortunately were able to convince the higher ups that this person was an imposter and we got rid of them. I really hope that these type of situations won't increase because the mental strain that put on some people in the org is not sustainable in the long run.

booleandilemma 2y ago

People aren't dumb. They'll catch on pretty quick that this thing is BS'ing them.

ethbr1 2y ago

I get a good chuckle every morning when the "C3.ai" ad rolls on NPR. "Hallucination-free," indeed. Would love to know what actual, contractual guarantees they place around that.

farmdve 2y ago

Next it's going to teach them the Earth is flat and there are aliens behind the moon.

CWuestefeld 2y ago

I hear these complaints and can't see how this is worse than the pre-AI situation. How is an AI "hallucination" different from human-generated works that are just plain wrong, or otherwise misleading? Humans make mistakes all the time. Teachers certainly did back when I was in school. There's no fundamental qualitative difference here. And I don't even see any evidence that there's any difference in degree, either.

lnxg33k1 2y ago
Probably the main difference is that humans fail at smaller scale, with smaller effects, and build a reputation, probably chatgpt hallucinations can potentially affect everyone
unclebucknasty 2y ago
>There's no fundamental qualitative difference here...degree either. I've heard the same comparisons made with self-driving cars (i.e. that humans are fallible, and maybe even more error-prone). But this misses the point. People trust the fallibility they know. That is, we largely understand human failure modes (errors in judgement, lapses in attention, etc) and feel like we are in control of them (and we are). OTOH, when machines make mistakes, they are experienced as unpredictable and outside of our control. Additionally, our expectation of machines is that they are deterministic and not subject to mistakes. While we know bugs can exist, it's not the expectation. And, with the current generation of AI in particular, we are dealing with models that are generally probabilistic, which means there's not even the expectation that they are errorless. And, I don't believe it's reasonable to expect people to give up control to AI of this quality, particularly in matters of safety or life and death; really anything that matters. TLDR; Most people don't want to gamble their lives on a statistic, when the alternative is maintaining control.
- chaorace 2y ago
  Expanding on this, human failures and machine failures are qualitatively different in ways that make our systems generally less resilient against the machine variety, even when dealing with a theoretically near-perfect implementation. Consider a bug in an otherwise perfect self-driving car routine that causes crashes under a highly specific scenario -- roads are essentially static structures, so you've effectively concentrated 100% of crashes into (for example) 1% of corridors. Practically speaking, those corridors would be forced into a state of perpetual closure. This is all to say that randomly distributed failures are more tolerable than a relatively smaller number of concentrated failures. Human errors are rather nice by comparison because they're inconsistent in locality while still being otherwise predictable in macroscopic terms (e.g.: on any given day, there will always be far more rear-endings than head-on collisions). When it comes to machine networks, all it takes is one firmware update for both the type & locality of their failure modes to go into a wildly different direction.

Dumblydorr 2y ago

Surgeons don’t need a text based LLM to make decisions. They have a job to do and a dozen years of training into how to do it. They have 8 years of schooling and 4-6 years internship and residency. The tech fantasy that everyone is using these for everything is a bubble thought. I agree with another comment, this is Doomerism.

CuriouslyC 2y ago
Surgeons are using robots that are far beyond fly by wire though, to the point that you could argue they're instructing the robots rather than controlling them.

HarHarVeryFunny 2y ago

Israel is already doing exactly that... using AI to identify potential targets based on their network of connections, giving these potential targets a cursory human screening, then OK-ing the bombing of their entire family since they have put such faith (and/or just don't care) in this identification process that these are considered high-value targets where "collateral damage" is accepted.

deleted 2y ago
[deleted]

bobosha 2y ago

https://pessimistsarchive.org/ https://pessimistsarchive.org/

DeathArrow 2y ago

>Who is going to prevent this from being used to pick military targets When AI is in charge of controlling weapons, you get this: https://www.accessnow.org/publication/artificial-genocidal-intelligence-israel-gaza/ https://www.accessnow.org/publication/artificial-genocidal-i...

chazeon 2y ago

It seems US and China are trying to reach an agreement to use AI to pick military targets these days.

gdubs 2y ago

While this is clearly a problem and a challenge to address, the thing that never gets mentioned with this line of criticism is the obvious: a large number of real-life teachers make mistakes ALL the time. They harbor wrong / out-dated opinions, or they're just flat-out wrong about things.