4 ms·

Open sourcing the algorithms (however we define it) does absolutely nothing. What use is a neural network architecture? Or a trained NN with some weights? Or an

by s1t5 6y ago

Open sourcing the algorithms (however we define it) does absolutely nothing. What use is a neural network architecture? Or a trained NN with some weights? Or an explanation that says - we measure similar posts by this metric and after you click on something we start serving you similar posts? None of those things are secret. More transparency wouldn't change anything because even if completely different algorithms were used, the fundamental problems with the platform would be exactly the same.

banads 6y ago
It's silly to so confidently assert that opening up a closed source algorithm to 3rd party analysis will "do absolutely nothing". How could you possibly know there is nothing unusual in the code without having audited it yourself? Seeing how the sausage gets made certainly can make lots of people lose their taste for it.
- ChrisLomont 6y ago
  A lot of how big systems work is embodied in large neural networks, and the detailed structure of how they make decisions is an open research problem. So it’s not silly for OP to state that at all; it’s empirical fact. It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips, through the doping in the chips, and even lower. This stack is such a hard problem that DARPA has run programs for a long time to address this. Start by reading Thompson’s ACM article titled something like “Reflections on Trustimg Trust” where he shows code audits don’t catch program behavior, then follow the past few decades where these holes have been pushed through the entire computing stack.
  - banads 6y ago
    >It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips, through the doping in the chips, and even lower. Are you implying that since we can't audit every single thing, auditing anything is useless?
    - thejynxed 6y ago
      The entire stack contains enough holes as to be swiss cheese, auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. This is one of the reasons those big security issues in Intel CPUs the last few years were such a big deal. The entire stack needs to be reworked at this point.
      - ChrisLomont 6y ago
        There isn't a simple fix, or likely any "fix," for the issues you want to be knowable. Besides the economic impossibility of it, there are too many places to hide behavior that we cannot foresee due to quantum effects, complexity, etc. So reworking the entire stack is not reasonable or likely very beneficial. It's better to incrementally address issues as they are found and weighted.
      - banads 6y ago
        >auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. In terms of software security vulnerabilities, there is so much low hanging fruit making exploitation trivial. Even if a small team within an intelligence agency knows about a zero day deep in the stack, addressing vulnerabilities higher up in the stack that are easily exploited by script kiddies necessarily reduces attack surface. However, what we're talking about here is not so much about security vulnerabilities, as it is about design flaws (or features) which have harmful effects on society.
    - ChrisLomont 6y ago
      >Are you implying that since we can't audit every single thing, auditing anything is useless? No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. And most of the activity by big companies is not in code, it's in data, and "auditing" it is currently beyond anything on the near horizon. Some of the behavior is un-auditable in the Halting Problem sense - i.e., the things you'd want to know are non-computable.
  - devonkim 6y ago
    Toolchain compromises are a non-zero risk but involve a lot of orchestrated resources to subvert systems to meaningful effect (simple exfil is sufficient for most corporate espionage v. stuff like Stuxnet to enact specific changes covertly). A company doing that given legislation to keep recommendation behavior and policies transparent to the public would be violating the spirit of the regulation by creating more opacity and delusion, no question. Admittedly, they're not going to be prosecuted in our current regulatory cyberpunk-esque hellscape, but neither would any public-benefit regulation pass anyway making the discussion of subversion moot, right? So presuming such a societal environment where regulation _could_ pass, we would hopefully have a more effective regulatory policy framework where subversion of the intent to be transparent for the sake of public safety and trust while still protecting trade secrets would be under sufficient scrutiny. All I know is that engineer-activists like Jaron Lanier are working with more tech-aware activists / politicians like Yang in proposing more effective tech regulatory frameworks than the past, and their efforts should be a lot more effective than either the current collective actions of the throwing up of our hands or yelling, whining, and screaming hoarsely. From a regulatory standpoint mirroring the nature of our organizational tendencies, I posit that the _policy_ models should look similar to Mickens' security threat vector model - Not-Mossad or Mossad.

by s1t5 6y ago

banads 6y ago

It's silly to so confidently assert that opening up a closed source algorithm to 3rd party analysis will "do absolutely nothing". How could you possibly know there is nothing unusual in the code without having audited it yourself? Seeing how the sausage gets made certainly can make lots of people lose their taste for it.

ChrisLomont 6y ago
A lot of how big systems work is embodied in large neural networks, and the detailed structure of how they make decisions is an open research problem. So it’s not silly for OP to state that at all; it’s empirical fact. It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips, through the doping in the chips, and even lower. This stack is such a hard problem that DARPA has run programs for a long time to address this. Start by reading Thompson’s ACM article titled something like “Reflections on Trustimg Trust” where he shows code audits don’t catch program behavior, then follow the past few decades where these holes have been pushed through the entire computing stack.
- banads 6y ago
  >It’s also not possible to audit the code for anything unusual without taking it all the way back through all tools source, all hardware, down through the chips, through the doping in the chips, and even lower. Are you implying that since we can't audit every single thing, auditing anything is useless?
  - thejynxed 6y ago
    The entire stack contains enough holes as to be swiss cheese, auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. This is one of the reasons those big security issues in Intel CPUs the last few years were such a big deal. The entire stack needs to be reworked at this point.
    - ChrisLomont 6y ago
      There isn't a simple fix, or likely any "fix," for the issues you want to be knowable. Besides the economic impossibility of it, there are too many places to hide behavior that we cannot foresee due to quantum effects, complexity, etc. So reworking the entire stack is not reasonable or likely very beneficial. It's better to incrementally address issues as they are found and weighted.
    - banads 6y ago
      >auditing the open code means nothing if and when something before that code in the stack manipulates the outcome of the code. In terms of software security vulnerabilities, there is so much low hanging fruit making exploitation trivial. Even if a small team within an intelligence agency knows about a zero day deep in the stack, addressing vulnerabilities higher up in the stack that are easily exploited by script kiddies necessarily reduces attack surface. However, what we're talking about here is not so much about security vulnerabilities, as it is about design flaws (or features) which have harmful effects on society.
  - ChrisLomont 6y ago
    >Are you implying that since we can't audit every single thing, auditing anything is useless? No, I am pointing out that your statement implying one can know if there's anything unusual in the code can be found by an audit. It cannot. And most of the activity by big companies is not in code, it's in data, and "auditing" it is currently beyond anything on the near horizon. Some of the behavior is un-auditable in the Halting Problem sense - i.e., the things you'd want to know are non-computable.
- devonkim 6y ago
  Toolchain compromises are a non-zero risk but involve a lot of orchestrated resources to subvert systems to meaningful effect (simple exfil is sufficient for most corporate espionage v. stuff like Stuxnet to enact specific changes covertly). A company doing that given legislation to keep recommendation behavior and policies transparent to the public would be violating the spirit of the regulation by creating more opacity and delusion, no question. Admittedly, they're not going to be prosecuted in our current regulatory cyberpunk-esque hellscape, but neither would any public-benefit regulation pass anyway making the discussion of subversion moot, right? So presuming such a societal environment where regulation _could_ pass, we would hopefully have a more effective regulatory policy framework where subversion of the intent to be transparent for the sake of public safety and trust while still protecting trade secrets would be under sufficient scrutiny. All I know is that engineer-activists like Jaron Lanier are working with more tech-aware activists / politicians like Yang in proposing more effective tech regulatory frameworks than the past, and their efforts should be a lot more effective than either the current collective actions of the throwing up of our hands or yelling, whining, and screaming hoarsely. From a regulatory standpoint mirroring the nature of our organizational tendencies, I posit that the _policy_ models should look similar to Mickens' security threat vector model - Not-Mossad or Mossad.