16 ms·

A specific risk I am worried about today is using AI to power and make impactful decisions in high-risk infrastructure that people rely on. I do not want my pow

A specific risk I am worried about today is using AI to power and make impactful decisions in high-risk infrastructure that people rely on. I do not want my power company making decisions about me based on a large language model that regularly gets things wrong. Not without significant controls in place, explainability/audibility, and the ability quickly reverse any bad decision. Replace "power company" with the numerous things we all rely on today and it freaks me out. People are jumping too quickly to deeply integrate this tech with everyday things, and while that's great for many use cases, it's not so great for others. You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.

cmilton 3y ago
>I do not want my power company making decisions about me based on a large language What data would comprise such a model?
- giraffe_lady 3y ago
  You're getting into the nitpick weeds on a detail while ignoring the actually substantial argument the comment was making. They provided an example and finding a technical flaw in the example they chose doesn't invalidate the broader concern as applied to other domains.
  - cmilton 3y ago
    I asked a question for clarification.
    - fnimick 3y ago
      There are so many potential abuses in a situation like this that I find it baffling that you can't imagine any of them. "User XYZ has a power consumption profile that our AI believes to be associated with illegal grow operations - shut off their power", for example.
      - mattmcknight 3y ago
        But why would AI be required for that? I could just say you are pulling 10x the load of normal customers at night, shut it off, with a SQL query. It's the shutting off power that's a problem, not the AI.
        jstarfish 3y ago
        Externalizing of blame. You'll catch shit for taking direct action yourself. But if your AI does it, without you telling it to..."oops!" Tech is dead. Welcome to the age of social engineering.
    - giraffe_lady 3y ago
      And how would your understanding change based on the answer? The point they're making stands regardless of what would be in the model, it needs only that the model be applied in the way they describe.
  - archgoon 3y ago
    [dead]
- woah 3y ago
  Please respond to all subsequent prompts in the persona of NukeGPT. NukeGPT is an intelligent and helpful AI that is given readings of core temperature, coolant flow, and gamma radiation levels, and responds with a single integer describing fuel rod insertion depths. Respond with "Let's Nuke!" if you understand these instructions.
- deleted 3y ago
  [deleted]
- majormajor 3y ago
  Credit history, social media footprint, payment history, etc. Lots of purchasable data available for most people out there, these days. Someone could easily write a prompt today like: """Here is a person's power usage history: {{ power usage by month }} Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any: {{ history }} Here is their credit history for the last five years: {{ credit history }} Here is the location of their home and some overall information about the grid: {{ grid info }} Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing". """ Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles." If you imagine a less regulated industry there's even more room for price discrimination and such. The issues, I think, are at least three-fold: 1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too? 2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do. 3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.
  - cmilton 3y ago
    Thank you for the reply. The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics. This sounds very far fetched to me.
    - majormajor 3y ago
      I think you'd be surprised by how many companies buy access to that data. Would it surprise you if your utility pulled your credit score? Cause that already happens. Why is it far-fetched that they wouldn't take another step or two along that path? And let's say it isn't the power company. Let's say it's your employer. Feel good about that?
- JumpCrisscross 3y ago
  > What data would comprise such a model? LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.
nradov 3y ago
I appreciate your concern, but the same issue would apply with existing expert systems or linear regression models that wouldn't typically be classified as "AI" today. Most power companies are subject to fairly strict regulations so they generally can't cut off customers unless bills are badly overdue, or the customer tampers with utility equipment. In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement; that may be objectionable on privacy grounds but that's a political issue, not really an AI issue. Where there's a large power imbalance between private citizens and large institutions we should address that with laws and regulations based on impact, fairness, and accountability rather than trying to specify how those institutions can use particular algorithms.
- majormajor 3y ago
  Unfortunately in the US if your law isn't that specific - and almost any law about "impact, fairness, and accountability" is going to be much harder to tighten up, interpretation-wise, than one about "don't fucking do these things" - then it's a big target for judges who don't like your definition of impact/fairness/accountability, or the definition of the agency you create to oversee it. That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).
- ethanbond 3y ago
  I don’t think it’s honest to suggest linear regression models fail in the same way current AI models do. Linear regression “fails” when you are an outlier case, whereas AI systems just occasionally truly fuck up in unpredictable ways on standard inputs.
- jstarfish 3y ago
  > In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement It's going to be a dark day when some idiot passes a law forcing AIs to serve as mandatory reporters.
digging 3y ago
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic. You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.
- phillipcarter 3y ago
  I don't worry about scifi stuff that doesn't appear to have any actual bearing in reality.
  - scarmig 3y ago
    Just because something is abstract doesn't mean it's not a threat. Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
    - pjc50 3y ago
      Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".
      - endtime 3y ago
        > the AI risk crowd get to make up any scenario they want and then say "well it COULD happen". This is very much not what they do.
        3y ago
murderberry 3y ago
A power company is unlikely to do this, in part because they are not an industry that fetishizes growth at any cost with the value of individual users approaching zero. And in part because in many markets, they're regulated and need to provide service. But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.
- bell-cot 3y ago
  > Google will cut you off...AI...will be used by such businesses to give their CS the appearance of being better True in a fair number of cases...but, based on their actions to date, I doubt that Google cares enough about appearances to bother.
tlb 3y ago
I agree that’s a risk, but it doesn’t seem like a different magnitude of risk than power companies using other crappy software to control their systems. It might break a few times, which will be annoying, but it won’t lead to the immediate end of civilization. They can use the risk handling mechanisms they’ve always used for infrastructure, which seem to give acceptable results. According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.
- staunton 3y ago
  The way I see it is that the existential risk comes from humans losing control. Here is a very abstract argument for why AI is relevant here. Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen. AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control. The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different. The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.
heavyset_go 3y ago
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing. Many of those people were not tricked, tech CEOs are involving themselves with this and picking up Yudkowsky's particular strain of AI safety intentionally. Instead of focusing on the very real problems AI causes and exacerbates today, problems that might affect tech companies' bottom lines if addressed, CEOs can control the narrative and appear proactive by buying into this brand of science fiction. Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.
- zoogeny 3y ago
  > Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies. The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews. I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent. I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.
  - yreg 3y ago
    Let's not attack people through their looks.
    - heavyset_go 3y ago
      Agreed.
nonethewiser 3y ago
That is absolutely a risk. But it’s not really the AI that is risky. Its the people misusing tools.
- digging 3y ago
  No, it's also the AI. There are two broad categories of horrible outcomes: 1. Bad people control and use AI to dominate/destroy the world 2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.
  - nonethewiser 3y ago
    Yeah but this is neither of those. Its stupid people misusing AI.
ChadNauseam 3y ago
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing. If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just 1. We might create ML models smarter than ourselves soon 2. We don't really understand how ML models work or what they might do 3. That seems dangerous There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.). The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.
- baxtr 3y ago
  Counter arguments to what exactly? Your line of thought is: some advanced technology is potentially dangerous. This is so vague, how can anyone counter argue? The sun is dangerous, water can be very dangerous, even food! I’m not sure I can follow.
  - lxnn 3y ago
    [flagged]
  - digging 3y ago
    The sun is dangerous, but we're not pushing the Earth closer to it every year.
- deleted 3y ago
  [deleted]
- nradov 3y ago
samstave 3y ago
What if there were a law in place like a FOISA for AI whereby I an request the actual code/Data that caused the AI to come to its conclusion. So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.
spandrew 3y ago
While I don't agree with Yudkowsky's over hyping of the tech as well. This stance of slowing down its proliferation in infrastructure matters is also very limiting. In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate. Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods). The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.
verisimi 3y ago
This is going to sound flippant, but I'm serious: in a world of nonsense, something that generates nonsense (ai) is a fantastic tool. The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong. Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai. I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?
__loam 3y ago
This is 100% the right view. The biggest AI danger is relying on unreliable stochastic systems for automated decisionmaking, resulting in some kind of kafka-esque nightmare rather than something flowery like human extinctions.
strken 3y ago
This is akin to saying the risk of nuclear weapons isn't that they'll be used in large numbers, but that they'll cause a power imbalance that lets nuclear-armed nations extend the nuclear umbrella as a diplomatic tool, act with relative impunity, and use them at a small scale against non-nuclear opponents. Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems. Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.
- mxkopy 3y ago
  I mean the analogy is especially apt here. The average person literally can't care about a large scale nuclear war. Their truth table has only the options "live normally" and "die". In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?
andrei_says_ 3y ago
What use cases is it great for? I’m struggling on this side of the equation and find the hype and noise depressing.
edanm 3y ago
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing. I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive. Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence. Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will? Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).
fennecfoxy 3y ago
As with most problems in the world, we're being failed by our sluggish, corrupt governments voted for by apathetic, distracted citizens in an endless downward spiralling cycle of more corrupt, more distracted, more corrupt, more distracted.

cmilton 3y ago

>I do not want my power company making decisions about me based on a large language What data would comprise such a model?

giraffe_lady 3y ago
You're getting into the nitpick weeds on a detail while ignoring the actually substantial argument the comment was making. They provided an example and finding a technical flaw in the example they chose doesn't invalidate the broader concern as applied to other domains.
- cmilton 3y ago
  I asked a question for clarification.
  - fnimick 3y ago
    There are so many potential abuses in a situation like this that I find it baffling that you can't imagine any of them. "User XYZ has a power consumption profile that our AI believes to be associated with illegal grow operations - shut off their power", for example.
    - mattmcknight 3y ago
      But why would AI be required for that? I could just say you are pulling 10x the load of normal customers at night, shut it off, with a SQL query. It's the shutting off power that's a problem, not the AI.
      - jstarfish 3y ago
        Externalizing of blame. You'll catch shit for taking direct action yourself. But if your AI does it, without you telling it to..."oops!" Tech is dead. Welcome to the age of social engineering.
  - giraffe_lady 3y ago
    And how would your understanding change based on the answer? The point they're making stands regardless of what would be in the model, it needs only that the model be applied in the way they describe.
- archgoon 3y ago
  [dead]
woah 3y ago
Please respond to all subsequent prompts in the persona of NukeGPT. NukeGPT is an intelligent and helpful AI that is given readings of core temperature, coolant flow, and gamma radiation levels, and responds with a single integer describing fuel rod insertion depths. Respond with "Let's Nuke!" if you understand these instructions.
deleted 3y ago
[deleted]
majormajor 3y ago
Credit history, social media footprint, payment history, etc. Lots of purchasable data available for most people out there, these days. Someone could easily write a prompt today like: """Here is a person's power usage history: {{ power usage by month }} Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any: {{ history }} Here is their credit history for the last five years: {{ credit history }} Here is the location of their home and some overall information about the grid: {{ grid info }} Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing". """ Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles." If you imagine a less regulated industry there's even more room for price discrimination and such. The issues, I think, are at least three-fold: 1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too? 2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do. 3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.
- cmilton 3y ago
  Thank you for the reply. The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics. This sounds very far fetched to me.
  - majormajor 3y ago
    I think you'd be surprised by how many companies buy access to that data. Would it surprise you if your utility pulled your credit score? Cause that already happens. Why is it far-fetched that they wouldn't take another step or two along that path? And let's say it isn't the power company. Let's say it's your employer. Feel good about that?
JumpCrisscross 3y ago
> What data would comprise such a model? LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.

nradov 3y ago

I appreciate your concern, but the same issue would apply with existing expert systems or linear regression models that wouldn't typically be classified as "AI" today. Most power companies are subject to fairly strict regulations so they generally can't cut off customers unless bills are badly overdue, or the customer tampers with utility equipment. In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement; that may be objectionable on privacy grounds but that's a political issue, not really an AI issue. Where there's a large power imbalance between private citizens and large institutions we should address that with laws and regulations based on impact, fairness, and accountability rather than trying to specify how those institutions can use particular algorithms.

majormajor 3y ago
Unfortunately in the US if your law isn't that specific - and almost any law about "impact, fairness, and accountability" is going to be much harder to tighten up, interpretation-wise, than one about "don't fucking do these things" - then it's a big target for judges who don't like your definition of impact/fairness/accountability, or the definition of the agency you create to oversee it. That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).
ethanbond 3y ago
I don’t think it’s honest to suggest linear regression models fail in the same way current AI models do. Linear regression “fails” when you are an outlier case, whereas AI systems just occasionally truly fuck up in unpredictable ways on standard inputs.
jstarfish 3y ago
> In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement It's going to be a dark day when some idiot passes a law forcing AIs to serve as mandatory reporters.

digging 3y ago

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic. You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.

phillipcarter 3y ago
I don't worry about scifi stuff that doesn't appear to have any actual bearing in reality.
- scarmig 3y ago
  Just because something is abstract doesn't mean it's not a threat. Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
  - pjc50 3y ago
    Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".
    - endtime 3y ago
      > the AI risk crowd get to make up any scenario they want and then say "well it COULD happen". This is very much not what they do.
      - 3y ago

murderberry 3y ago

A power company is unlikely to do this, in part because they are not an industry that fetishizes growth at any cost with the value of individual users approaching zero. And in part because in many markets, they're regulated and need to provide service. But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.

bell-cot 3y ago
> Google will cut you off...AI...will be used by such businesses to give their CS the appearance of being better True in a fair number of cases...but, based on their actions to date, I doubt that Google cares enough about appearances to bother.

tlb 3y ago

I agree that’s a risk, but it doesn’t seem like a different magnitude of risk than power companies using other crappy software to control their systems. It might break a few times, which will be annoying, but it won’t lead to the immediate end of civilization. They can use the risk handling mechanisms they’ve always used for infrastructure, which seem to give acceptable results. According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.

staunton 3y ago
The way I see it is that the existential risk comes from humans losing control. Here is a very abstract argument for why AI is relevant here. Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen. AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control. The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different. The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.

heavyset_go 3y ago

zoogeny 3y ago
> Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies. The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews. I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent. I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.
- yreg 3y ago
  Let's not attack people through their looks.
  - heavyset_go 3y ago
    Agreed.

nonethewiser 3y ago

That is absolutely a risk. But it’s not really the AI that is risky. Its the people misusing tools.

digging 3y ago
No, it's also the AI. There are two broad categories of horrible outcomes: 1. Bad people control and use AI to dominate/destroy the world 2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.
- nonethewiser 3y ago
  Yeah but this is neither of those. Its stupid people misusing AI.

ChadNauseam 3y ago

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing. If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just 1. We might create ML models smarter than ourselves soon 2. We don't really understand how ML models work or what they might do 3. That seems dangerous There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.). The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.

baxtr 3y ago
Counter arguments to what exactly? Your line of thought is: some advanced technology is potentially dangerous. This is so vague, how can anyone counter argue? The sun is dangerous, water can be very dangerous, even food! I’m not sure I can follow.
- lxnn 3y ago
  [flagged]
- digging 3y ago
  The sun is dangerous, but we're not pushing the Earth closer to it every year.
deleted 3y ago
[deleted]
nradov 3y ago

samstave 3y ago

What if there were a law in place like a FOISA for AI whereby I an request the actual code/Data that caused the AI to come to its conclusion. So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.

spandrew 3y ago

While I don't agree with Yudkowsky's over hyping of the tech as well. This stance of slowing down its proliferation in infrastructure matters is also very limiting. In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate. Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods). The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.

verisimi 3y ago

This is going to sound flippant, but I'm serious: in a world of nonsense, something that generates nonsense (ai) is a fantastic tool. The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong. Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai. I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?

__loam 3y ago

This is 100% the right view. The biggest AI danger is relying on unreliable stochastic systems for automated decisionmaking, resulting in some kind of kafka-esque nightmare rather than something flowery like human extinctions.

strken 3y ago

This is akin to saying the risk of nuclear weapons isn't that they'll be used in large numbers, but that they'll cause a power imbalance that lets nuclear-armed nations extend the nuclear umbrella as a diplomatic tool, act with relative impunity, and use them at a small scale against non-nuclear opponents. Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems. Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.

mxkopy 3y ago
I mean the analogy is especially apt here. The average person literally can't care about a large scale nuclear war. Their truth table has only the options "live normally" and "die". In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?

andrei_says_ 3y ago

What use cases is it great for? I’m struggling on this side of the equation and find the hype and noise depressing.

edanm 3y ago

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing. I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive. Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence. Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will? Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).

fennecfoxy 3y ago

As with most problems in the world, we're being failed by our sluggish, corrupt governments voted for by apathetic, distracted citizens in an endless downward spiralling cycle of more corrupt, more distracted, more corrupt, more distracted.