3 ms·

Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figur

by mcbits 4y ago

Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? I think if you go into class and ask the teacher how to build a bomb, they will refuse to answer, but not because they were never allowed to learn enough about the world to figure it out for themselves. Just about anyone with a basic education can figure it out if they're put to the task, but that doesn't mean the parents and school administrators want it taught in the classroom. The teacher also had to learn the concept of "a device that explodes with great force and is typically used to harm other people" somewhere in order to recognize that they're not supposed to present those kinds of instructions in general. And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society.

tablespoon 4y ago
> Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? Probably a decent amount, honestly, and certainly enough for a "high school ... introductory physics or chemistry class." I mean, what does high school chemistry cover? Balancing chemical equations, Arrhenius acids and bases, basic lab work, and other very basic stuff like that? Someone can excel in the class by carefully reading a textbook that certainly doesn't cover bomb-making. It's not like students are going into novel applications or anything. > Just about anyone with a basic education can figure it out if they're put to the task... Without a recipe or off-the-shelf explosives? I doubt it. Or we're a nation of MacGyvers and I had no idea. And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. These models can't figure anything out, they just regurgitate and associate on command. There's no reason they need to have inappropriate content irreverent to their purpose available to regurgitate. Leaving a student who wants to build a pipe bomb in the situation of having to put themselves to the task of figuring it out from the basics themselves, instead of having their homework-help model effortlessly regurgitate it for them is fine. > And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society. To be blunt, I think you're going off into la la land following generalizations and tangents.
- mcbits 4y ago
  > Balancing chemical equations, Arrhenius acids and bases, basic lab work, and ... ... oxidation, exothermic reactions? A teacher who never actually studied the subject and only has knowledge of the pre-filtered material from the textbook would be little more than a talking textbook. I mean, textbooks are useful, but we already have those. > Without a recipe or off-the-shelf explosives? If they've ever studied the relevant subjects, yes. > And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. But the suggestion to pre-filter is more like a teacher who was never allowed to learn enough to succeed at the task even if they tried. Teachers who could succeed but refuse to try are the norm. (Or maybe they have tried on their own time, or done the thought experiment, but they know that handing out step-by-step instructions would be taboo in most circumstances.) > These models can't figure anything out, they just regurgitate and associate on command. I think you may be overemphasizing "regurgitate" and underemphasizing "associate". ChatGPT can emit instructions for just about anything that were clearly not just regurgitated from some other source, and surprisingly often they're even accurate - or accurate enough to use as a starting point. Let's just agree that pre-filtering the training text is not a practical way to prevent these models from producing emissions that are considered dangerous or taboo, at least not without losing the traits that make them so interesting in the first place. But the self-censoring hidden prompts that they're using can also lead to absurd results, so there's a lot of room for improvement. I was trying to generate a fictional story where at one point the main character tries walk out of a store before paying for an item, and it refused to write something "illegal or unethical". It was okay with having the character leave some money on the counter before walking out.
  - tablespoon 4y ago
    >> Balancing chemical equations, Arrhenius acids and bases, basic lab work, and ... > ... oxidation, exothermic reactions? So? The syllabus doesn't include the chemistry of actual explosives or bomb design. Heck, it probably include little if any practical chemistry. It's all basic, basic, basic concepts. Look, I could do a decent job helping a high school kid with their Chemistry homework, including things like redox reactions and exothermic reactions without even thinking about anything related to bombs. All those concepts can and are taught with "safer" and simpler reactions. > A teacher who never actually studied the subject and only has knowledge of the pre-filtered material from the textbook would be little more than a talking textbook. I mean, textbooks are useful, but we already have those. God help us if some dumb GPT model is supposed to replace teachers. There's a whole lot of empirical knowledge actual practicing chemists have to learn from reports or find out experimentally, but don't know off the top of their (e.g. investigations about particular reactions, including particular explosives). If actual competent people don't know that, a stupid model for high school students doesn't need to be trained on it. Similar with practical bomb-making info. You haven't provided any compelling argument for inclusion of bomb-relevant material in training data besides appealing to vague analogies. > Let's just agree that pre-filtering the training text is not a practical way to prevent these models from producing emissions that are considered dangerous or taboo... Sorry, I can't exactly agree with that. While pre-filtering might not be capable of wholly preventing all undesirable outputs, I think it can make them less likely and far less useful. If a high school student trying to make a pipe bomb has to feed in all the conceptual and practical information about bombs and explosives, and gets some impractical reaction out of it. That's a win, because it means the model wasn't much if any help. And there's precedent for that. I'm under the impression that later versions of Stable Diffusion are pretty bad at making porn, because porn and porn-related keywords were pre-filtered better: https://www.theverge.com/2022/11/24/23476622/ai-image-generator-stable-diffusion-version-2-nsfw-artists-data-changes https://www.theverge.com/2022/11/24/23476622/ai-image-genera....

by mcbits 4y ago

tablespoon 4y ago

> Well, let's put it another way. How much chemistry or physics education can you get from a teacher who has never learned enough about their own subject to figure out how to build something like a pipe bomb? Probably a decent amount, honestly, and certainly enough for a "high school ... introductory physics or chemistry class." I mean, what does high school chemistry cover? Balancing chemical equations, Arrhenius acids and bases, basic lab work, and other very basic stuff like that? Someone can excel in the class by carefully reading a textbook that certainly doesn't cover bomb-making. It's not like students are going into novel applications or anything. > Just about anyone with a basic education can figure it out if they're put to the task... Without a recipe or off-the-shelf explosives? I doubt it. Or we're a nation of MacGyvers and I had no idea. And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. These models can't figure anything out, they just regurgitate and associate on command. There's no reason they need to have inappropriate content irreverent to their purpose available to regurgitate. Leaving a student who wants to build a pipe bomb in the situation of having to put themselves to the task of figuring it out from the basics themselves, instead of having their homework-help model effortlessly regurgitate it for them is fine. > And this is just an example. The same goes for thousands of other topics in hundreds of other fields that would be inappropriate for teachers to teach. If you shelter them all from the world to the point that they're completely ignorant of the forbidden topics, they would be ineffective as teachers. A school full of those teachers would be ineffective as a school. A society filled with such schools would be stunted as a society. To be blunt, I think you're going off into la la land following generalizations and tangents.

mcbits 4y ago
> Balancing chemical equations, Arrhenius acids and bases, basic lab work, and ... ... oxidation, exothermic reactions? A teacher who never actually studied the subject and only has knowledge of the pre-filtered material from the textbook would be little more than a talking textbook. I mean, textbooks are useful, but we already have those. > Without a recipe or off-the-shelf explosives? If they've ever studied the relevant subjects, yes. > And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher. But the suggestion to pre-filter is more like a teacher who was never allowed to learn enough to succeed at the task even if they tried. Teachers who could succeed but refuse to try are the norm. (Or maybe they have tried on their own time, or done the thought experiment, but they know that handing out step-by-step instructions would be taboo in most circumstances.) > These models can't figure anything out, they just regurgitate and associate on command. I think you may be overemphasizing "regurgitate" and underemphasizing "associate". ChatGPT can emit instructions for just about anything that were clearly not just regurgitated from some other source, and surprisingly often they're even accurate - or accurate enough to use as a starting point. Let's just agree that pre-filtering the training text is not a practical way to prevent these models from producing emissions that are considered dangerous or taboo, at least not without losing the traits that make them so interesting in the first place. But the self-censoring hidden prompts that they're using can also lead to absurd results, so there's a lot of room for improvement. I was trying to generate a fictional story where at one point the main character tries walk out of a store before paying for an item, and it refused to write something "illegal or unethical". It was okay with having the character leave some money on the counter before walking out.
- tablespoon 4y ago
  >> Balancing chemical equations, Arrhenius acids and bases, basic lab work, and ... > ... oxidation, exothermic reactions? So? The syllabus doesn't include the chemistry of actual explosives or bomb design. Heck, it probably include little if any practical chemistry. It's all basic, basic, basic concepts. Look, I could do a decent job helping a high school kid with their Chemistry homework, including things like redox reactions and exothermic reactions without even thinking about anything related to bombs. All those concepts can and are taught with "safer" and simpler reactions. > A teacher who never actually studied the subject and only has knowledge of the pre-filtered material from the textbook would be little more than a talking textbook. I mean, textbooks are useful, but we already have those. God help us if some dumb GPT model is supposed to replace teachers. There's a whole lot of empirical knowledge actual practicing chemists have to learn from reports or find out experimentally, but don't know off the top of their (e.g. investigations about particular reactions, including particular explosives). If actual competent people don't know that, a stupid model for high school students doesn't need to be trained on it. Similar with practical bomb-making info. You haven't provided any compelling argument for inclusion of bomb-relevant material in training data besides appealing to vague analogies. > Let's just agree that pre-filtering the training text is not a practical way to prevent these models from producing emissions that are considered dangerous or taboo... Sorry, I can't exactly agree with that. While pre-filtering might not be capable of wholly preventing all undesirable outputs, I think it can make them less likely and far less useful. If a high school student trying to make a pipe bomb has to feed in all the conceptual and practical information about bombs and explosives, and gets some impractical reaction out of it. That's a win, because it means the model wasn't much if any help. And there's precedent for that. I'm under the impression that later versions of Stable Diffusion are pretty bad at making porn, because porn and porn-related keywords were pre-filtered better: https://www.theverge.com/2022/11/24/23476622/ai-image-generator-stable-diffusion-version-2-nsfw-artists-data-changes https://www.theverge.com/2022/11/24/23476622/ai-image-genera....