2 ms·

> I think most free software folks would agree that the training data doesn't have to be public. That seems antithetical to me. The trained model is the progra

by gnode 7y ago

> I think most free software folks would agree that the training data doesn't have to be public. That seems antithetical to me. The trained model is the program, and the training data is in-essence the source code. > manipulating what an AI does by manipulating the training data You don't even have to do this. You can manipulate the model directly. Writing neural networks manually isn't difficult. You can't do more with that than conventional programming, but its sufficient to add malicious behaviour.

kerkeslager 7y ago
> That seems antithetical to me. The trained model is the program, and the training data is in-essence the source code. No, I think that trained models and training data are entirely new concepts. I can see the similarities between trained models/programs a and training data/source code, but there are obvious privacy concerns with training data that don't apply to source code. Calling the training data the source code is a leaky abstraction. You could, for example, equally argue that the trained model is the source code, and the program which operates on the trained model is an interpreter which runs that source code. Even if you refuse to concede that there are differences between training data and source code, you'll note that I am talking about free (libre) software, not open source. There are lots of cases where free software doesn't mean opening all the source up. Free software is about freedom, not about open source. For example, while there's plenty of source code in settings files, I don't think there's anyone arguing that we need to check in our settings files on open source projects. The point is to put the power in the hands of users. Putting users' data out of their control is the antithesis of that. > You don't even have to do this. You can manipulate the model directly. Writing neural networks manually isn't difficult. You can't do more with that than conventional programming, but its sufficient to add malicious behaviour. That's an interesting point, and may provide a workaround making training data public, i.e. you can generate the model and then make a representation of the model public, rather than the training data. However, there's an assumption here that the training data can't be reconstructed from the trained model--if a study of this exists, I'm not aware of it.
- gnode 7y ago
  I think the trained model is analogous to a program (which may be interpreted by a virtual machine; not necessarily a machine code program) because it is not intelligible by humans. I'll admit that there are tools for analysing neural networks, but these are like disassemblers. Trained models carry all the hazards of binary blobs, and so can't just be trusted. Training data is no true analogue to source code, but the concept of reproducibility is still relevant. At the very least, the production of blobs should be auditable.
  - kerkeslager 7y ago
    > I think the trained model is analogous to a program (which may be interpreted by a virtual machine; not necessarily a machine code program) because it is not intelligible by humans. I'll admit that there are tools for analysing neural networks, but these are like disassemblers. Yeah, I agree that's a valid analogy, but again I'll say it's just an analogy. The training data still differs from source code in its privacy attributes, so I don't think it should be treated the same way as source code in a free (libre) software context. > Trained models carry all the hazards of binary blobs, and so can't just be trusted. Training data is no true analogue to source code, but the concept of reproducibility is still relevant. At the very least, the production of blobs should be auditable. This is true, and it's a big problem, not just because of free software concerns. One of the deeply problematic effects of this is that we can't always explain why an AI does what it does, leading to horrifying possibilities[1]. [1] https://www.jwz.org/blog/2019/04/frogger-ai-explains-its-decisions/ https://www.jwz.org/blog/2019/04/frogger-ai-explains-its-dec... (WARNING: I recommend copy/pasting this link to a separate tab rather than clicking it. JWZ has a rather negative opinion of HN and a while back his server started serving up rather unsavory images in response to requests with an HN referrer. I'm not sure if that is still happening, but I know that if you copy/paste the link there's nothing offensive on the page.)

by gnode 7y ago

kerkeslager 7y ago

> That seems antithetical to me. The trained model is the program, and the training data is in-essence the source code. No, I think that trained models and training data are entirely new concepts. I can see the similarities between trained models/programs a and training data/source code, but there are obvious privacy concerns with training data that don't apply to source code. Calling the training data the source code is a leaky abstraction. You could, for example, equally argue that the trained model is the source code, and the program which operates on the trained model is an interpreter which runs that source code. Even if you refuse to concede that there are differences between training data and source code, you'll note that I am talking about free (libre) software, not open source. There are lots of cases where free software doesn't mean opening all the source up. Free software is about freedom, not about open source. For example, while there's plenty of source code in settings files, I don't think there's anyone arguing that we need to check in our settings files on open source projects. The point is to put the power in the hands of users. Putting users' data out of their control is the antithesis of that. > You don't even have to do this. You can manipulate the model directly. Writing neural networks manually isn't difficult. You can't do more with that than conventional programming, but its sufficient to add malicious behaviour. That's an interesting point, and may provide a workaround making training data public, i.e. you can generate the model and then make a representation of the model public, rather than the training data. However, there's an assumption here that the training data can't be reconstructed from the trained model--if a study of this exists, I'm not aware of it.

gnode 7y ago
I think the trained model is analogous to a program (which may be interpreted by a virtual machine; not necessarily a machine code program) because it is not intelligible by humans. I'll admit that there are tools for analysing neural networks, but these are like disassemblers. Trained models carry all the hazards of binary blobs, and so can't just be trusted. Training data is no true analogue to source code, but the concept of reproducibility is still relevant. At the very least, the production of blobs should be auditable.
- kerkeslager 7y ago
  > I think the trained model is analogous to a program (which may be interpreted by a virtual machine; not necessarily a machine code program) because it is not intelligible by humans. I'll admit that there are tools for analysing neural networks, but these are like disassemblers. Yeah, I agree that's a valid analogy, but again I'll say it's just an analogy. The training data still differs from source code in its privacy attributes, so I don't think it should be treated the same way as source code in a free (libre) software context. > Trained models carry all the hazards of binary blobs, and so can't just be trusted. Training data is no true analogue to source code, but the concept of reproducibility is still relevant. At the very least, the production of blobs should be auditable. This is true, and it's a big problem, not just because of free software concerns. One of the deeply problematic effects of this is that we can't always explain why an AI does what it does, leading to horrifying possibilities[1]. [1] https://www.jwz.org/blog/2019/04/frogger-ai-explains-its-decisions/ https://www.jwz.org/blog/2019/04/frogger-ai-explains-its-dec... (WARNING: I recommend copy/pasting this link to a separate tab rather than clicking it. JWZ has a rather negative opinion of HN and a while back his server started serving up rather unsavory images in response to requests with an HN referrer. I'm not sure if that is still happening, but I know that if you copy/paste the link there's nothing offensive on the page.)