7 ms·

The Coming AI Hackers (2021)

70 points by josefslerka 4y ago

amelius 4y ago
> But some bugs in the tax code are also vulnerabilities. For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.” So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do?
- pcthrowaway 4y ago
  Computer hackers don't necessarily do jail time. Someone hacking a single-player video game, to give them increased stats or something, won't do jail time. Even when it's a multiplayer game, someone finding a hack that benefits them is unlikely to face jail time. Hacking is a crime if you're gaining unauthorized access to a system, especially a remote one. I think hacks involving money and personal user data have different criminal classifications as well. And that's ignoring the massive number of hackers ("researchers") who actually get paid to do it (many of whom started out doing illegal things).
  - amelius 4y ago
    Yes, but a hacking and making millions is generally a crime, while exploiting the tax code to make millions or even billions is accepted behavior.
    - grenoire 4y ago
      As if there's an implicit belief that companies artificially reducing their tax liability will lead to better growth or something? Perceived benefit is then positive, I suppose. On the other hand, an individual doing it for their personal gain is vilified indeed.
- xwdv 4y ago
  You don’t understand, when it comes to taxes we follow the letter of the law, not the spirit. Financial hackers are technically using legal means to accomplish their goals. The law is the law.
  - amelius 4y ago
    My point is that this is strange. Breaking into a government website is illegal. However if the code on their server was part of "the law", it suddenly becomes legal?
    - xwdv 4y ago
      If the law says it’s legal, that is the law.
      - amelius 4y ago
        Still strange. It should be: if it's against the spirit of the law, it's illegal. Otherwise we could replace judges by computers who just evaluate logic statements in the book of law.
        xwdv 4y ago
        No, fuck the spirit of the law. If that’s what society runs itself on then the written laws don’t matter and the real laws become whatever subjective bullshit the judges decide at the moment. “Hidden” laws essentially. Dystopia. Logic statements that evaluate to legal or illegal is the ideal when it comes to laws, judges are only necessary when logical certainty isn’t possible.
        amelius 4y ago
        I see a contradiction there. You want the law to be like code, but actual computer code you are still not allowed to use in unintended ways? How would that last part be encoded in the law? Anyway, I'm ok with a more logical, objective interpretation of the law, but then also the loopholes in the law should be treated as code exploits: it should be possible to report them and then they should be fixed asap.
- jason-phillips 4y ago
  If you mean "exploiting" loopholes, then they are lawful and usually working as intended. I never considered my CPA a criminal hacker; I'm sure he'll be amused. If you simply mean "not paying your taxes" then this is a crime that is frequently prosecuted and is not particularly clever or a hack. You appear to be conflating the two for some reason. If you mean, "unauthorized access of a computer system", then yes, that is unlawful and would possibly be one reason why some people are prosecuted and sent to jail. Nit: >> For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.” This has not been a thing since 2020.
  - amelius 4y ago
    > You appear to be conflating the two for some reason. No, I'm just abstractly (like TFA) considering hacking to be an activity where a system (code or financial law) is brought outside of its intended use. > "exploiting" loopholes Anyway, why aren't there bounties for finding loopholes and why aren't they fixed as soon as possible?
    - jason-phillips 4y ago
      Because the loopholes are a feature. Working as intended in the vast majority of instances.
    - palata 4y ago
      You seem to consider that the law is defined by the system being hacked. It is not: the law is the law. The code of a website is not the law. It is legal to hack a system if it does not break the law (I can use a fork as a screwdriver if I want to), but it is illegal if it breaks the law (I can use the fork as a weapon and break the law). A loophole is legal. It may feel like it should not be, because it allows to work around the law. But it is legal.
- drdec 4y ago
  > So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do? That's because Schneier defines hacking in the original sense of the word, subverting or misusing a system to get a desired, but unintended, result. The computer "hackers" doing jail time are not hackers by Scheier's definition.
nonrandomstring 4y ago
Normally I finish Bruce's essays with a sense of clarity and having read a sharp analysis. Not saying there's anything technically amis with this one, it's all terrifyingly clear, but I think he bit off a little more that he could chew. The result is indigestible. I personally would have split this into several shorter pieces. When it's all put together this way though, I can't help think we've shot ourselves in the foot. As computer scientists and developers we've probably already lost control. We owe a debt of honesty to the world to say so now and stop pretending otherwise. Then we can, as a society, revisit Postman's Seven Questions: 1. What is the problem that this new technology solves? 2. Whose problem is it? 3. What new problems do we create by solving this problem? 4. Which people and institutions will be most impacted by a technological solution? 5. What changes in language occur as the result of technological change? 6. Which shifts in economic and political power might result when this technology is adopted? 7. What alternative (and unintended) uses might be made of this technology?
- JKCalhoun 4y ago
  Thanks, I had not seen these Seven Questions before. As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators. * Well, at best it would devolve into arguing back and forth and neither side able to convince the other what problems would or would not be caused or are avoidable.
  - privong 4y ago
    > As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators. That is likely true in many cases, but, even if definitive answers cannot be provided, the questions should still be asked in advance and the answers pondered. Many people take that difficulty or impossibility of conclusive answers as justification for not asking those later questions.
  - 4y ago
jillesvangurp 4y ago
Interesting, but long read. To summarize: people are easy to manipulate and AIs are learning how to do this at scale and with intent. AIs are still tools though. It's easy to anthropomorphize AIs and think in terms of us and them. It's how we are wired to think. But the reality is that they are expensive tools that are owned and wielded by other people. So, you get all the unimaginative dystopian thinking. Which is of course largely the product of science fiction written decades ago. It's hard to be innovative in this space. We already imagined all the bad outcomes a long time ago. But that doesn't mean it will play out like that. Precisely because we've imagined all the bad stuff, that's likely to not happen. The short term reality is actually an arms race between companies and countries. And like most arms races, the people with the best toys and the most resources end up on top. The question is not what AIs will do but what the people wielding them will do with them. And how we can hack their purposes. The system these companies and countries operate in is of course hack-able. Democracies are a power hack. We once had these all powerful kings, emperors, and dictators. And then the people that gave them power got smart and organized. Democracies basically curb that power in the interest of self preservation. Like with most technology, the answer to what will happen with AIs will be mostly harmless and beneficial stuff. That's where the money is. But with some intentionally harmful stuff and maybe a little bit of unintentionally disastrous stuff. So, we as a society need to get better at countering/preventing/disincentivizing the bad stuff and preventing the unintentionally harmful stuff. Rejecting technology is not the answer. It just makes us more vulnerable. The more of us get smarter, the better it is for all of us.
- jcq3 4y ago
  This was summarized by chatGPT*
- palata 4y ago
  I mean no offence at all, but it somehow feels like you don't know who the author is. Doesn't mean you cannot disagree of course, but I believe his opinion deserves credit :-). The second thing is that I strongly disagree with your statement that technology is mostly harmless and beneficial. In one sentence: it is responsible for the current mass extinction. But of course there are tons of small examples that show how technology is generally not "mostly harmless and beneficial". I would even argue that blockchain, for instance, has turned out to be mostly harmful: I am still looking for a constructive use-case of it, though criminal ones are legion. ChatGPT, for instance, will bring phishing/impersonation/disinformation to a whole new level. That should be scary, given the impact of less elaborate disinformation technologies in the past.
machina_ex_deus 4y ago
This article is written as if scheier is going to make collective decisions on behalf of everyone, or as if someone making collective decisions for everyone is going to stop everything "because it's dangerous". What's actually going to happen is that regardless of the so called possible dangers, there will be talented humans who still view this as a net positive thing and they will continue to develop AI, wherever it goes. Unless there's a global traumatic event, I don't see anything stopping. AI developers have an interest to fake moral problems to better capitalize on their product (we don't support you running models at home, it's too dangerous), and that's exactly where their ethics concerns will end. We can't deal with any of these things by telling people to stop doing things they obviously won't stop doing. I'd rather we deal with this transparently. How about requiring AI models deployed to be really open? Then at least humanity will always have an advantage over AI that AIs are whitebox while humans are blackbox. And if some AI will do some hack to benefit its creators, everyone else will have a chance to understand it's happening. It shouldn't be "this is dangerous, make it closed so only we can abuse it", it should be, this is dangerous, so we're doing this as transparently as possible. I think in a theoretical war between Turing machines, the Turing machine which is given as input the code of the other turing machine should always be the winner. Think about the halting problem and diagonalization: the diagonalizing counter example wins by having the source code of the program which supposedly solves the halting problem.
- CatWChainsaw 4y ago
  Or we could have a Butlerian Jihad. :)
- palata 4y ago
  Did you read the part about the Explainability Problem? I think it says why it is fundamentally not transparent.
williamcotton 4y ago
It seems inevitable that we will need some kind of PKE in order to verify identities. My ideal system is using the CAC protocol from the DoD at each of the 50 DMVs in the US to issue identification. I don’t think this form of state-issued PKE should be required for getting online but I would prefer interacting with people in an environment where they were needed to participate. Of course anonymous forums should be allowed but I don’t want anonymity in every interaction… …especially over the next few years as an increasing number of actors will be non-human…
YeGoblynQueenne 4y ago
>> This hacking will come naturally, as AIs become more advanced at learning, understanding, and problem-solving. Everyone is so sure "AI"s will continue improving (not to say, everyone is so sure they have improved). It's going to be a little embarrassing if the foretold continuous AI improvement does not come to pass. :grabs popcorn:
- palata 4y ago
  I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm. It can write disinformation at an incredible scale. Now whenever you visit a website, just ask yourself this question: "was it written by a person, or by a bot?". It's getting hard to say. Of course it does not always matter. But intuitively, wouldn't you put more trust into a website written by a human that actually understand what they wrote? "Visiting Prague in 3 days" written by a human may not be perfect, but at least the human understand the idea of "visiting a city in 3 days". Now think about actual disinformation. Can you still put any trust in an article not written on a website you know (e.g. The Guardian)? It was already a problem before, but with AI like GPT, it is now possible to "flood" the web with disinformation written in pretty high quality language...
  - YeGoblynQueenne 4y ago
    >> I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm. Absolutely, but that is because it hasn't "advanced" as much as some people seem to think. Others seem to see the problems but they seem to hope or believe that they will all go away as those systems "advance" (or "improve" or "progress") further, when they may as well just have reached a dead end after which they can only become bigger and more bloated as time goes by, and still be dumb as bricks, and the only thing that will "advance" is their ability to cause chaos and make a mess. I like this quote by Pedro Domingos: >> People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world. https://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-domingos-author-of-the-master-algorithm/ https://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-...
nocsi 4y ago
DARPA had a program some years back to train AI on actual hackers to model their behaviors. Anyways, I still doubt AI's ability to cause significant harm beyond automating spear-phishing, social engineering, mass-scanning codebases/binaries and exploiting low-hanging fruits. Modern day exploits are still outside the realm of what AI can do - and I don't even believe AI will ever exceed that realm. But I think society will do a better job w/ security once AI automates and annoys everyone
roxgib 4y ago
> First, participants interacted with the robot in a normal setting to experience its performance, which was deliberately poor. Then, they had to decide whether or not to follow the robot’s commands in a simulated emergency. In the latter situation, all twenty-six participants obeyed the robot, despite having observed just moments before that the robot had lousy navigational skills. The degree of trust they placed in this machine was striking: when the robot pointed to a dark room with no clear exit, the majority of people obeyed it, rather than safely exiting by the door through which they had entered. I see this all the time - people putting their faith in systems and rules and computer programs despite knowing that they're more than likely wrong in the given situation. It's bizarre.

70 points by josefslerka 4y ago

> But some bugs in the tax code are also vulnerabilities. For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.” So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do?

pcthrowaway 4y ago
Computer hackers don't necessarily do jail time. Someone hacking a single-player video game, to give them increased stats or something, won't do jail time. Even when it's a multiplayer game, someone finding a hack that benefits them is unlikely to face jail time. Hacking is a crime if you're gaining unauthorized access to a system, especially a remote one. I think hacks involving money and personal user data have different criminal classifications as well. And that's ignoring the massive number of hackers ("researchers") who actually get paid to do it (many of whom started out doing illegal things).
- amelius 4y ago
  Yes, but a hacking and making millions is generally a crime, while exploiting the tax code to make millions or even billions is accepted behavior.
  - grenoire 4y ago
    As if there's an implicit belief that companies artificially reducing their tax liability will lead to better growth or something? Perceived benefit is then positive, I suppose. On the other hand, an individual doing it for their personal gain is vilified indeed.
xwdv 4y ago
You don’t understand, when it comes to taxes we follow the letter of the law, not the spirit. Financial hackers are technically using legal means to accomplish their goals. The law is the law.
- amelius 4y ago
  My point is that this is strange. Breaking into a government website is illegal. However if the code on their server was part of "the law", it suddenly becomes legal?
  - xwdv 4y ago
    If the law says it’s legal, that is the law.
    - amelius 4y ago
      Still strange. It should be: if it's against the spirit of the law, it's illegal. Otherwise we could replace judges by computers who just evaluate logic statements in the book of law.
      - xwdv 4y ago
        No, fuck the spirit of the law. If that’s what society runs itself on then the written laws don’t matter and the real laws become whatever subjective bullshit the judges decide at the moment. “Hidden” laws essentially. Dystopia. Logic statements that evaluate to legal or illegal is the ideal when it comes to laws, judges are only necessary when logical certainty isn’t possible.
        amelius 4y ago
        I see a contradiction there. You want the law to be like code, but actual computer code you are still not allowed to use in unintended ways? How would that last part be encoded in the law? Anyway, I'm ok with a more logical, objective interpretation of the law, but then also the loopholes in the law should be treated as code exploits: it should be possible to report them and then they should be fixed asap.
jason-phillips 4y ago
If you mean "exploiting" loopholes, then they are lawful and usually working as intended. I never considered my CPA a criminal hacker; I'm sure he'll be amused. If you simply mean "not paying your taxes" then this is a crime that is frequently prosecuted and is not particularly clever or a hack. You appear to be conflating the two for some reason. If you mean, "unauthorized access of a computer system", then yes, that is unlawful and would possibly be one reason why some people are prosecuted and sent to jail. Nit: >> For example, there’s a corporate tax trick called the “Double Irish with a Dutch Sandwich.” This has not been a thing since 2020.
- amelius 4y ago
  > You appear to be conflating the two for some reason. No, I'm just abstractly (like TFA) considering hacking to be an activity where a system (code or financial law) is brought outside of its intended use. > "exploiting" loopholes Anyway, why aren't there bounties for finding loopholes and why aren't they fixed as soon as possible?
  - jason-phillips 4y ago
    Because the loopholes are a feature. Working as intended in the vast majority of instances.
  - palata 4y ago
    You seem to consider that the law is defined by the system being hacked. It is not: the law is the law. The code of a website is not the law. It is legal to hack a system if it does not break the law (I can use a fork as a screwdriver if I want to), but it is illegal if it breaks the law (I can use the fork as a weapon and break the law). A loophole is legal. It may feel like it should not be, because it allows to work around the law. But it is legal.
drdec 4y ago
> So how come financial hackers exploiting vulnerabilities in the tax code and making these sandwiches never do jail time, while computer hackers regularly do? That's because Schneier defines hacking in the original sense of the word, subverting or misusing a system to get a desired, but unintended, result. The computer "hackers" doing jail time are not hackers by Scheier's definition.

nonrandomstring 4y ago

Normally I finish Bruce's essays with a sense of clarity and having read a sharp analysis. Not saying there's anything technically amis with this one, it's all terrifyingly clear, but I think he bit off a little more that he could chew. The result is indigestible. I personally would have split this into several shorter pieces. When it's all put together this way though, I can't help think we've shot ourselves in the foot. As computer scientists and developers we've probably already lost control. We owe a debt of honesty to the world to say so now and stop pretending otherwise. Then we can, as a society, revisit Postman's Seven Questions: 1. What is the problem that this new technology solves? 2. Whose problem is it? 3. What new problems do we create by solving this problem? 4. Which people and institutions will be most impacted by a technological solution? 5. What changes in language occur as the result of technological change? 6. Which shifts in economic and political power might result when this technology is adopted? 7. What alternative (and unintended) uses might be made of this technology?

JKCalhoun 4y ago
Thanks, I had not seen these Seven Questions before. As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators. * Well, at best it would devolve into arguing back and forth and neither side able to convince the other what problems would or would not be caused or are avoidable.
- privong 4y ago
  > As a pragmatist (?), I suspect answering from 3) on down is near impossible* unless you actually release it into the world. People are clever in a way I don't think any of us can anticipate and put to use things in novel ways that often surprise their creators. That is likely true in many cases, but, even if definitive answers cannot be provided, the questions should still be asked in advance and the answers pondered. Many people take that difficulty or impossibility of conclusive answers as justification for not asking those later questions.
- 4y ago

jillesvangurp 4y ago

Interesting, but long read. To summarize: people are easy to manipulate and AIs are learning how to do this at scale and with intent. AIs are still tools though. It's easy to anthropomorphize AIs and think in terms of us and them. It's how we are wired to think. But the reality is that they are expensive tools that are owned and wielded by other people. So, you get all the unimaginative dystopian thinking. Which is of course largely the product of science fiction written decades ago. It's hard to be innovative in this space. We already imagined all the bad outcomes a long time ago. But that doesn't mean it will play out like that. Precisely because we've imagined all the bad stuff, that's likely to not happen. The short term reality is actually an arms race between companies and countries. And like most arms races, the people with the best toys and the most resources end up on top. The question is not what AIs will do but what the people wielding them will do with them. And how we can hack their purposes. The system these companies and countries operate in is of course hack-able. Democracies are a power hack. We once had these all powerful kings, emperors, and dictators. And then the people that gave them power got smart and organized. Democracies basically curb that power in the interest of self preservation. Like with most technology, the answer to what will happen with AIs will be mostly harmless and beneficial stuff. That's where the money is. But with some intentionally harmful stuff and maybe a little bit of unintentionally disastrous stuff. So, we as a society need to get better at countering/preventing/disincentivizing the bad stuff and preventing the unintentionally harmful stuff. Rejecting technology is not the answer. It just makes us more vulnerable. The more of us get smarter, the better it is for all of us.

jcq3 4y ago
This was summarized by chatGPT*
palata 4y ago
I mean no offence at all, but it somehow feels like you don't know who the author is. Doesn't mean you cannot disagree of course, but I believe his opinion deserves credit :-). The second thing is that I strongly disagree with your statement that technology is mostly harmless and beneficial. In one sentence: it is responsible for the current mass extinction. But of course there are tons of small examples that show how technology is generally not "mostly harmless and beneficial". I would even argue that blockchain, for instance, has turned out to be mostly harmful: I am still looking for a constructive use-case of it, though criminal ones are legion. ChatGPT, for instance, will bring phishing/impersonation/disinformation to a whole new level. That should be scary, given the impact of less elaborate disinformation technologies in the past.

machina_ex_deus 4y ago

This article is written as if scheier is going to make collective decisions on behalf of everyone, or as if someone making collective decisions for everyone is going to stop everything "because it's dangerous". What's actually going to happen is that regardless of the so called possible dangers, there will be talented humans who still view this as a net positive thing and they will continue to develop AI, wherever it goes. Unless there's a global traumatic event, I don't see anything stopping. AI developers have an interest to fake moral problems to better capitalize on their product (we don't support you running models at home, it's too dangerous), and that's exactly where their ethics concerns will end. We can't deal with any of these things by telling people to stop doing things they obviously won't stop doing. I'd rather we deal with this transparently. How about requiring AI models deployed to be really open? Then at least humanity will always have an advantage over AI that AIs are whitebox while humans are blackbox. And if some AI will do some hack to benefit its creators, everyone else will have a chance to understand it's happening. It shouldn't be "this is dangerous, make it closed so only we can abuse it", it should be, this is dangerous, so we're doing this as transparently as possible. I think in a theoretical war between Turing machines, the Turing machine which is given as input the code of the other turing machine should always be the winner. Think about the halting problem and diagonalization: the diagonalizing counter example wins by having the source code of the program which supposedly solves the halting problem.

CatWChainsaw 4y ago
Or we could have a Butlerian Jihad. :)
palata 4y ago
Did you read the part about the Explainability Problem? I think it says why it is fundamentally not transparent.

williamcotton 4y ago

It seems inevitable that we will need some kind of PKE in order to verify identities. My ideal system is using the CAC protocol from the DoD at each of the 50 DMVs in the US to issue identification. I don’t think this form of state-issued PKE should be required for getting online but I would prefer interacting with people in an environment where they were needed to participate. Of course anonymous forums should be allowed but I don’t want anonymity in every interaction… …especially over the next few years as an increasing number of actors will be non-human…

YeGoblynQueenne 4y ago

>> This hacking will come naturally, as AIs become more advanced at learning, understanding, and problem-solving. Everyone is so sure "AI"s will continue improving (not to say, everyone is so sure they have improved). It's going to be a little embarrassing if the foretold continuous AI improvement does not come to pass. :grabs popcorn:

palata 4y ago
I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm. It can write disinformation at an incredible scale. Now whenever you visit a website, just ask yourself this question: "was it written by a person, or by a bot?". It's getting hard to say. Of course it does not always matter. But intuitively, wouldn't you put more trust into a website written by a human that actually understand what they wrote? "Visiting Prague in 3 days" written by a human may not be perfect, but at least the human understand the idea of "visiting a city in 3 days". Now think about actual disinformation. Can you still put any trust in an article not written on a website you know (e.g. The Guardian)? It was already a problem before, but with AI like GPT, it is now possible to "flood" the web with disinformation written in pretty high quality language...
- YeGoblynQueenne 4y ago
  >> I do agree, though it feels like ChatGPT, for instance, already has a lot of potential to harm. Absolutely, but that is because it hasn't "advanced" as much as some people seem to think. Others seem to see the problems but they seem to hope or believe that they will all go away as those systems "advance" (or "improve" or "progress") further, when they may as well just have reached a dead end after which they can only become bigger and more bloated as time goes by, and still be dumb as bricks, and the only thing that will "advance" is their ability to cause chaos and make a mess. I like this quote by Pedro Domingos: >> People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world. https://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-domingos-author-of-the-master-algorithm/ https://www.washington.edu/news/2015/09/17/a-q-a-with-pedro-...

nocsi 4y ago

DARPA had a program some years back to train AI on actual hackers to model their behaviors. Anyways, I still doubt AI's ability to cause significant harm beyond automating spear-phishing, social engineering, mass-scanning codebases/binaries and exploiting low-hanging fruits. Modern day exploits are still outside the realm of what AI can do - and I don't even believe AI will ever exceed that realm. But I think society will do a better job w/ security once AI automates and annoys everyone

roxgib 4y ago

> First, participants interacted with the robot in a normal setting to experience its performance, which was deliberately poor. Then, they had to decide whether or not to follow the robot’s commands in a simulated emergency. In the latter situation, all twenty-six participants obeyed the robot, despite having observed just moments before that the robot had lousy navigational skills. The degree of trust they placed in this machine was striking: when the robot pointed to a dark room with no clear exit, the majority of people obeyed it, rather than safely exiting by the door through which they had entered. I see this all the time - people putting their faith in systems and rules and computer programs despite knowing that they're more than likely wrong in the given situation. It's bizarre.