4 ms·

> In an API we sanitize input/out programmatically. Are they not doing that? I think the problem is they're probably not adequately sanitizing the training inp

> In an API we sanitize input/out programmatically. Are they not doing that? I think the problem is they're probably not adequately sanitizing the training input. I'm no expert, but my understanding is that these things need to be trained on utterly massive amounts of text. I bet the most economical way by far to acquire that volume of text (especially conversational text) is to just feed in a web scrape that has a lot of forum, chatroom, and social media insanity. Then they try to kludge acceptability by trying to filter the output, but that's as ineffective as trying to validate by excluding invalid values rather than only allowing valid values. If their training data only had people asking respectable questions and responding professionally, I don't think there's any way you could get a rude or offensive response out of one of these things.

mcbits 4y ago
You can't train a language model capable of helping a high school student pass an introductory physics or chemistry class if the training data was filtered to the extent that the same language model can't explain how to build a pipe bomb. If you don't want it explaining how to build pipe bombs, either the filtering has to come later or you have to accept that the model will be useless for tons of innocent things that users would obviously want it to do.
- tablespoon 4y ago
  > You can't train a language model capable of helping a high school student pass an introductory physics or chemistry class if the training data was filtered to the extent that the same language model can't explain how to build a pipe bomb. Why not? I'm certain that "introductory physics or chemistry class" doesn't cover pipe bomb construction. There's literally no reason the model needs to even have the vocabulary term "pipe bomb" or all kinds of practical information required to construct one (e.g. explosive recipes, exact chemical make up of household chemicals, impurities and all, etc). I see a point that restricting training data of such a model might not be able to prevent it from being used to help with subtasks for "inventing" a pipe bomb (e.g. could you suggest a reactions between solids that releases a large amount of potential energy very quickly), but it feels like you'd already have to know so much about pipe bombs at that point to build one yourself without help. And even then, I'm not sure restricting training data couldn't help. Why train the thing on the reactions used in explosives? Why not leave a gap there, so the only reactions it knows about are only as energetic as the ones that would be performed in a classroom?
  - mcbits 4y ago
    Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? I think if you go into class and ask the teacher how to build a bomb, they will refuse to answer, but not because they were never allowed to learn enough about the world to figure it out for themselves. Just about anyone with a basic education can figure it out if they're put to the task, but that doesn't mean the parents and school administrators want it taught in the classroom. The teacher also had to learn the concept of "a device that explodes with great force and is typically used to harm other people" somewhere in order to recognize that they're not supposed to present those kinds of instructions in general. And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society.
    - tablespoon 4y ago
      > Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? Probably a decent amount, honestly, and certainly enough for a "high school ... introductory physics or chemistry class." I mean, what does high school chemistry cover? Balancing chemical equations, Arrhenius acids and bases, basic lab work, and other very basic stuff like that? Someone can excel in the class by carefully reading a textbook that certainly doesn't cover bomb-making. It's not like students are going into novel applications or anything. > Just about anyone with a basic education can figure it out if they're put to the task... Without a recipe or off-the-shelf explosives? I doubt it. Or we're a nation of MacGyvers and I had no idea. And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. These models can't figure anything out, they just regurgitate and associate on command. There's no reason they need to have inappropriate content irreverent to their purpose available to regurgitate. Leaving a student who wants to build a pipe bomb in the situation of having to put themselves to the task of figuring it out from the basics themselves, instead of having their homework-help model effortlessly regurgitate it for them is fine. > And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society. To be blunt, I think you're going off into la la land following generalizations and tangents.

mcbits 4y ago

You can't train a language model capable of helping a high school student pass an introductory physics or chemistry class if the training data was filtered to the extent that the same language model can't explain how to build a pipe bomb. If you don't want it explaining how to build pipe bombs, either the filtering has to come later or you have to accept that the model will be useless for tons of innocent things that users would obviously want it to do.

tablespoon 4y ago
> You can't train a language model capable of helping a high school student pass an introductory physics or chemistry class if the training data was filtered to the extent that the same language model can't explain how to build a pipe bomb. Why not? I'm certain that "introductory physics or chemistry class" doesn't cover pipe bomb construction. There's literally no reason the model needs to even have the vocabulary term "pipe bomb" or all kinds of practical information required to construct one (e.g. explosive recipes, exact chemical make up of household chemicals, impurities and all, etc). I see a point that restricting training data of such a model might not be able to prevent it from being used to help with subtasks for "inventing" a pipe bomb (e.g. could you suggest a reactions between solids that releases a large amount of potential energy very quickly), but it feels like you'd already have to know so much about pipe bombs at that point to build one yourself without help. And even then, I'm not sure restricting training data couldn't help. Why train the thing on the reactions used in explosives? Why not leave a gap there, so the only reactions it knows about are only as energetic as the ones that would be performed in a classroom?
- mcbits 4y ago
  Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? I think if you go into class and ask the teacher how to build a bomb, they will refuse to answer, but not because they were never allowed to learn enough about the world to figure it out for themselves. Just about anyone with a basic education can figure it out if they're put to the task, but that doesn't mean the parents and school administrators want it taught in the classroom. The teacher also had to learn the concept of "a device that explodes with great force and is typically used to harm other people" somewhere in order to recognize that they're not supposed to present those kinds of instructions in general. And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society.
  - tablespoon 4y ago
    > Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? Probably a decent amount, honestly, and certainly enough for a "high school ... introductory physics or chemistry class." I mean, what does high school chemistry cover? Balancing chemical equations, Arrhenius acids and bases, basic lab work, and other very basic stuff like that? Someone can excel in the class by carefully reading a textbook that certainly doesn't cover bomb-making. It's not like students are going into novel applications or anything. > Just about anyone with a basic education can figure it out if they're put to the task... Without a recipe or off-the-shelf explosives? I doubt it. Or we're a nation of MacGyvers and I had no idea. And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. These models can't figure anything out, they just regurgitate and associate on command. There's no reason they need to have inappropriate content irreverent to their purpose available to regurgitate. Leaving a student who wants to build a pipe bomb in the situation of having to put themselves to the task of figuring it out from the basics themselves, instead of having their homework-help model effortlessly regurgitate it for them is fine. > And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society. To be blunt, I think you're going off into la la land following generalizations and tangents.