5 ms·

>It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips

by banads 6y ago

>It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips, through the doping in the chips, and even lower. Are you implying that since we can't audit every single thing, auditing anything is useless?

thejynxed 6y ago
The entire stack contains enough holes as to be swiss cheese, auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. This is one of the reasons those big security issues in Intel CPUs the last few years were such a big deal. The entire stack needs to be reworked at this point.
- ChrisLomont 6y ago
  There isn't a simple fix, or likely any "fix," for the issues you want to be knowable. Besides the economic impossibility of it, there are too many places to hide behavior that we cannot foresee due to quantum effects, complexity, etc. So reworking the entire stack is not reasonable or likely very beneficial. It's better to incrementally address issues as they are found and weighted.
- banads 6y ago
  >auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. In terms of software security vulnerabilities, there is so much low hanging fruit making exploitation trivial. Even if a small team within an intelligence agency knows about a zero day deep in the stack, addressing vulnerabilities higher up in the stack that are easily exploited by script kiddies necessarily reduces attack surface. However, what we're talking about here is not so much about security vulnerabilities, as it is about design flaws (or features) which have harmful effects on society.
ChrisLomont 6y ago
>Are you implying that since we can't audit every single thing, auditing anything is useless? No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. And most of the activity by big companies is not in code, it's in data, and "auditing" it is currently beyond anything on the near horizon. Some of the behavior is un-auditable in the Halting Problem sense - i.e., the things you'd want to know are non-computable.
- banads 6y ago
  >No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. This is so plainly false its silly. Have you ever heard of a code review? What do you think security researchers do? Google Project Zero. Plenty of things are found all the time at the higher (and lower) levels of the stack, even if something unknown remains deep within. >And most of the activity by big companies is not in code, it's in data it is currently beyond anything on the near horizon. Audits have no problem finding out what type of data is being collected (see PCI || HIPAA compliance). That would be a great start: for people to be made explicitly aware of all the data points that are being collected on them.
  - ChrisLomont 6y ago
    >This is so plainly false its silly. You're simply wrong. Did you read the article that shows quite clearly exactly how to do this I told you about? No, you didn't, or you'd stop making this false claim. Before you repeat this, RTFA, which I'll post again since you didn't learn last time [1]. There, read it? There's decades of research into even deeper, more sophisticated ways to hide behavior. At the lowest level, against a malicious actor, there is no current way to ensure lack of bad behavior. >What do you think security researchers do? Yes, I've worked on security research projects for decades, winning millions of dollars for govt projects to do so. I am quite aware of the state of the art. You don't seem to be aware of basic things decades old. Do you actually work in security research? >Plenty of things are found all the time You're confusing finding accidental bugs with an actor trying to hide behavior. The latter you will not find if the actor is as big as a FAANG or nation state. If simply looking at things was sufficient, then the DoD wouldn't be afraid of Chinese made software or chips - they could simply look, right? But they know this is a fool's errand. They spend literally billions working this problem, year in and year out, for decades. It's naive that you think simple audits will root out bad behavior against malicious actors. Even accidental bugs live in huge, opensource projects for decades, passing audit after audit, only to be exploited decades later. These are accidental. How many could an actor like NSA implant with their resources that would survive your audits? Oh, did I mention [1]? Read it again. Read followup papers. Do some original research in this vein, and write some papers. Give talks on security about these techniques to other researchers. I've done all that. I have a pretty good idea how this works. >what type of data is being collected Again, you miss. I am not talking about the data being collected. I'm talking about the data in big systems that make decisions. NNs and all sorts of other AI-ish systems run huge parts of all the big companies, and these cannot yet be understood - it is literally an open research problem. Check DoD SBIR lists for the many, many places they're paying to have researchers (me, for example - I write proposals for this money) to help solve this problem. For the tip of the iceberg, read on adversarial image recognition and the arms race to detect or prevent it and how deep that rabbit hole goes. Now audit my image classifier and tell me which adversarial image systems it is weak against. Tell me if I embedded any adversarial behavior into it. Oh yeah, you cannot do either, because it's an unsolved (and possibly unsolvable) problem. Now do this for every learning system, such as Amazon's recommender system, for Facebook's ad placement algorithms, for Google's search results. You literally cannot. Don't bother replying until you understand the paper - it shows that a code audit will not turn up malicious behavior if the owner is actively trying to hide stuff from you. [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

by banads 6y ago

thejynxed 6y ago

The entire stack contains enough holes as to be swiss cheese, auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. This is one of the reasons those big security issues in Intel CPUs the last few years were such a big deal. The entire stack needs to be reworked at this point.

ChrisLomont 6y ago
There isn't a simple fix, or likely any "fix," for the issues you want to be knowable. Besides the economic impossibility of it, there are too many places to hide behavior that we cannot foresee due to quantum effects, complexity, etc. So reworking the entire stack is not reasonable or likely very beneficial. It's better to incrementally address issues as they are found and weighted.
banads 6y ago
>auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. In terms of software security vulnerabilities, there is so much low hanging fruit making exploitation trivial. Even if a small team within an intelligence agency knows about a zero day deep in the stack, addressing vulnerabilities higher up in the stack that are easily exploited by script kiddies necessarily reduces attack surface. However, what we're talking about here is not so much about security vulnerabilities, as it is about design flaws (or features) which have harmful effects on society.

ChrisLomont 6y ago

>Are you implying that since we can't audit every single thing, auditing anything is useless? No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. And most of the activity by big companies is not in code, it's in data, and "auditing" it is currently beyond anything on the near horizon. Some of the behavior is un-auditable in the Halting Problem sense - i.e., the things you'd want to know are non-computable.

banads 6y ago
>No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. This is so plainly false its silly. Have you ever heard of a code review? What do you think security researchers do? Google Project Zero. Plenty of things are found all the time at the higher (and lower) levels of the stack, even if something unknown remains deep within. >And most of the activity by big companies is not in code, it's in data it is currently beyond anything on the near horizon. Audits have no problem finding out what type of data is being collected (see PCI || HIPAA compliance). That would be a great start: for people to be made explicitly aware of all the data points that are being collected on them.
- ChrisLomont 6y ago
  >This is so plainly false its silly. You're simply wrong. Did you read the article that shows quite clearly exactly how to do this I told you about? No, you didn't, or you'd stop making this false claim. Before you repeat this, RTFA, which I'll post again since you didn't learn last time [1]. There, read it? There's decades of research into even deeper, more sophisticated ways to hide behavior. At the lowest level, against a malicious actor, there is no current way to ensure lack of bad behavior. >What do you think security researchers do? Yes, I've worked on security research projects for decades, winning millions of dollars for govt projects to do so. I am quite aware of the state of the art. You don't seem to be aware of basic things decades old. Do you actually work in security research? >Plenty of things are found all the time You're confusing finding accidental bugs with an actor trying to hide behavior. The latter you will not find if the actor is as big as a FAANG or nation state. If simply looking at things was sufficient, then the DoD wouldn't be afraid of Chinese made software or chips - they could simply look, right? But they know this is a fool's errand. They spend literally billions working this problem, year in and year out, for decades. It's naive that you think simple audits will root out bad behavior against malicious actors. Even accidental bugs live in huge, opensource projects for decades, passing audit after audit, only to be exploited decades later. These are accidental. How many could an actor like NSA implant with their resources that would survive your audits? Oh, did I mention [1]? Read it again. Read followup papers. Do some original research in this vein, and write some papers. Give talks on security about these techniques to other researchers. I've done all that. I have a pretty good idea how this works. >what type of data is being collected Again, you miss. I am not talking about the data being collected. I'm talking about the data in big systems that make decisions. NNs and all sorts of other AI-ish systems run huge parts of all the big companies, and these cannot yet be understood - it is literally an open research problem. Check DoD SBIR lists for the many, many places they're paying to have researchers (me, for example - I write proposals for this money) to help solve this problem. For the tip of the iceberg, read on adversarial image recognition and the arms race to detect or prevent it and how deep that rabbit hole goes. Now audit my image classifier and tell me which adversarial image systems it is weak against. Tell me if I embedded any adversarial behavior into it. Oh yeah, you cannot do either, because it's an unsolved (and possibly unsolvable) problem. Now do this for every learning system, such as Amazon's recommender system, for Facebook's ad placement algorithms, for Google's search results. You literally cannot. Don't bother replying until you understand the paper - it shows that a code audit will not turn up malicious behavior if the owner is actively trying to hide stuff from you. [1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...