4 ms·

>Do you actually work in security research? Yep! >You're confusing finding accidental bugs with an actor trying to hide behavior. The latter you will not find

by banads 6y ago

>Do you actually work in security research? Yep! >You're confusing finding accidental bugs with an actor trying to hide behavior. The latter you will not find if the actor is as big as a FAANG or nation state. Wrong, wrong, wrong. Here's but one of many examples: https://googleprojectzero.blogspot.com/2019/08/a-very-deep-dive-into-ios-exploit.html https://googleprojectzero.blogspot.com/2019/08/a-very-deep-d... >it shows that a code audit will not turn up malicious behavior if the owner is actively trying to hide stuff from you. Yes, code obfuscation is a thing, but it's not a perfect silver bullet like you falsely claim, and often only serves to slow down researchers. Of course it is true though that many bugs and vulnerabilities remain hidden which we may never find, and yet that's not a valid reason not to look for them, because there are many which are found every single day. The perfect is the enemy of the good.

ChrisLomont 6y ago
>> Do you actually work in security research? > Yep! Your comment history is interestingly lacking any evidence of that. Care to demonstrate it? >>The latter you will not find if the actor is ... > Wrong, wrong, wrong. Tell me how you can audit an image classifier to ensure it won't claim a specially marked tank is not a rabbit, where an enemy built the classifier, and where you don't know what markings the enemy will use. You're given the neural net code and all the weights, so you can run the system to your hearts content on your own hardware. Explain how to audit, please. Good luck. It's bizarre you claim people can find such things, when it's a huge current research problem, and it's open if such things can even be demonstrated at all. Same thing for literally any medium or large neural network. None can be audited for proof of no bad behavior. >code obfuscation is a thing, but it's not a perfect silver bullet like you falsely claim, I've never said code obfuscation. Unless you understand the paper which you seem to repeatedly ignore, you'll keep making the same mistake. The paper demonstrates how to remove the code that has the behavior while still embedding the bad behavior in the product. You cannot find it from auditing the product code. There is zero trace in the source code. The attack in the paper has been pushed through all layers of computing stack since then, and now is at the quantum level. And as these effects become more important, there can be no audit since what you want to know runs up against physics, such as the No Cloning Theorem. That you don't realize this is possible is why you keep making the same error that looking at source code will tell you what a product does. If your naive belief in this were that simple there would not be literally billions of dollars available for you to do what you claim you can. DoD/DARPA/Secure foundries would love to see your magic methods. Have you read the paper? Maybe it will show you the tip of the iceberg on why audits are used to find accidents, but are much weaker against adversarial actors, to the point of not providing any value for really complex systems. >The perfect is the enemy of the good. I'm not saying don't audit. I'm pointing out your initial claim that this will find anything unusual. It can find common errors. It can find bad behavior inserted by unskilled people. But against groups that know about current work, you won't find anything, in the same way you cannot audit a neural network. That you continue to think code obfuscation is the only way to embed bad behavior in a stack shows that you're unaware of a large section of security research. Read the paper.
- banads 6y ago
  >I'm not saying don't audit. I'm pointing out your initial claim that this will find anything unusual. By "unusual" I mean anything that has intended or unintended negative effects on society, such as what was seen with Cambridge Analytica, or FBs emotional manipulation studies. >It can find common errors. It can find bad behavior inserted by unskilled people. But against groups that know about current work, you won't find anything Yep, and without a 3rd party audit, we can't even begin to approximate the degree of hypothetical bad behavior that exists affecting billions of people due to regular developers doing what they're told by their product managers (or of their own volition), let alone a nation state APT.
  - ChrisLomont 6y ago
    You keep ignoring both how to audit NNs and how to address behavior not in code. Have you read the paper yet? Explain how audits work in light of the paper. Without answering those you’re simply wasting time and effort by claiming audits can find things they cannot. I quite doubt you work in security from your inability to grasp these things. Please demonstrate you’re not lying. Your posting history shows a tendency to be a conspiracy believer, and there’s zero evidence you do anything professionally in security, unlike the history of those I know that do work in security.

by banads 6y ago

ChrisLomont 6y ago

>> Do you actually work in security research? > Yep! Your comment history is interestingly lacking any evidence of that. Care to demonstrate it? >>The latter you will not find if the actor is ... > Wrong, wrong, wrong. Tell me how you can audit an image classifier to ensure it won't claim a specially marked tank is not a rabbit, where an enemy built the classifier, and where you don't know what markings the enemy will use. You're given the neural net code and all the weights, so you can run the system to your hearts content on your own hardware. Explain how to audit, please. Good luck. It's bizarre you claim people can find such things, when it's a huge current research problem, and it's open if such things can even be demonstrated at all. Same thing for literally any medium or large neural network. None can be audited for proof of no bad behavior. >code obfuscation is a thing, but it's not a perfect silver bullet like you falsely claim, I've never said code obfuscation. Unless you understand the paper which you seem to repeatedly ignore, you'll keep making the same mistake. The paper demonstrates how to remove the code that has the behavior while still embedding the bad behavior in the product. You cannot find it from auditing the product code. There is zero trace in the source code. The attack in the paper has been pushed through all layers of computing stack since then, and now is at the quantum level. And as these effects become more important, there can be no audit since what you want to know runs up against physics, such as the No Cloning Theorem. That you don't realize this is possible is why you keep making the same error that looking at source code will tell you what a product does. If your naive belief in this were that simple there would not be literally billions of dollars available for you to do what you claim you can. DoD/DARPA/Secure foundries would love to see your magic methods. Have you read the paper? Maybe it will show you the tip of the iceberg on why audits are used to find accidents, but are much weaker against adversarial actors, to the point of not providing any value for really complex systems. >The perfect is the enemy of the good. I'm not saying don't audit. I'm pointing out your initial claim that this will find anything unusual. It can find common errors. It can find bad behavior inserted by unskilled people. But against groups that know about current work, you won't find anything, in the same way you cannot audit a neural network. That you continue to think code obfuscation is the only way to embed bad behavior in a stack shows that you're unaware of a large section of security research. Read the paper.

banads 6y ago
>I'm not saying don't audit. I'm pointing out your initial claim that this will find anything unusual. By "unusual" I mean anything that has intended or unintended negative effects on society, such as what was seen with Cambridge Analytica, or FBs emotional manipulation studies. >It can find common errors. It can find bad behavior inserted by unskilled people. But against groups that know about current work, you won't find anything Yep, and without a 3rd party audit, we can't even begin to approximate the degree of hypothetical bad behavior that exists affecting billions of people due to regular developers doing what they're told by their product managers (or of their own volition), let alone a nation state APT.
- ChrisLomont 6y ago
  You keep ignoring both how to audit NNs and how to address behavior not in code. Have you read the paper yet? Explain how audits work in light of the paper. Without answering those you’re simply wasting time and effort by claiming audits can find things they cannot. I quite doubt you work in security from your inability to grasp these things. Please demonstrate you’re not lying. Your posting history shows a tendency to be a conspiracy believer, and there’s zero evidence you do anything professionally in security, unlike the history of those I know that do work in security.