3 ms·

There's a danger, while steelmanning, that you might come up with an argument that sounds good to you, but it's actually weaker to the opposition. Therefore, wh

There's a danger, while steelmanning, that you might come up with an argument that sounds good to you, but it's actually weaker to the opposition. Therefore, while trying to steelman their argument, you actually fill it with holes. This can happen because you don't understand it deeply enough, but it can also happen because of a difference in values. This is why I think steelmanning is difficult, and frankly a little dangerous. To truly do it, you have to embed yourself not just in their rational thought but also in their worldview. And I really mean the part about it being dangerous. In all honesty, we aren't the purely rational creatures we think we are. Honestly steelmanning an argument changes you. And maybe that's for the better in some situations, but I certainly wouldn't take that for granted.

nusaru 4y ago
Wouldn't the obvious solution be to steelman whatever you were steelmanning against in the first place?
antognini 4y ago
I think this gets at the related, but distinct, idea of the intellectual Turing test. The idea there is to present your opponent's argument in such a way that they can't tell if the argument came from you or someone on their side. If you can do that you have some assurance that you actually understand the argument they are making. You're right that steelmanning is a bit different in that you're trying to formulate the argument in a way that is most convincing to you. That may not necessarily be the argument that is most convincing to your opponent.
JetAlone 4y ago
Being able to steelman someone else's argument without adopting their presuppositions, while understanding and possible experiencing the emotional/intuitive phenomena that got them to embrace them is a true test of your faith in your own worldview. If you "fail" that test, you will be absorbed into the outgroup and lose sight of your old emotional/intuitive presuppositions and worldview. You will lose a lot of friends and sunk costs, and be subjected to new rules. The risk is to literally lose your old self. That's horrifying, that idea can feel something like being abducted by the borg from star trek and converted into one of them.
- FeepingCreature 4y ago
  Hence, keeping your identity small: http://www.paulgraham.com/identity.html http://www.paulgraham.com/identity.html
- XorNot 4y ago
  No, it isn't. Immersing yourself in someone else's propaganda just propagandizes you. The easiest victims are the ones who think they're genuinely too smart for it to have an effect on them. There's a reason country's pump this stuff out despite it seeming, on a cursory glance, so ridiculous and overt.
gwd 4y ago
> There's a danger, while steelmanning, that you might come up with an argument that sounds good to you, but it's actually weaker to the opposition. Therefore, while trying to steelman their argument, you actually fill it with holes. That's why normal practice is to check: "So I hear you saying something like this: <steelman argument>. Is that right?" > To truly do it, you have to embed yourself not just in their rational thought but also in their worldview. Indeed; the best word I have for being truly persuasive is "incarnational": You have to enter into the situation of the person you want to convince, and see the way out of the cave they're in from where they are now. > And I really mean the part about it being dangerous. In all honesty, we aren't the purely rational creatures we think we are. Honestly steelmanning an argument changes you. And maybe that's for the better in some situations, but I certainly wouldn't take that for granted. I dunno; I have more faith in the power of truth. You certainly need to be careful that you're not taking information only from one source; if you only listened to Goebel's propaganda for 2 years you'd certainly start to have Nazi patterns of thought infect your thinking. But if you want to rescue someone who has only heard Goebel's propaganda for the last several years, you have to learn how to recognize the flaws in thinking from within it.

nusaru 4y ago

Wouldn't the obvious solution be to steelman whatever you were steelmanning against in the first place?

antognini 4y ago

I think this gets at the related, but distinct, idea of the intellectual Turing test. The idea there is to present your opponent's argument in such a way that they can't tell if the argument came from you or someone on their side. If you can do that you have some assurance that you actually understand the argument they are making. You're right that steelmanning is a bit different in that you're trying to formulate the argument in a way that is most convincing to you. That may not necessarily be the argument that is most convincing to your opponent.

JetAlone 4y ago

Being able to steelman someone else's argument without adopting their presuppositions, while understanding and possible experiencing the emotional/intuitive phenomena that got them to embrace them is a true test of your faith in your own worldview. If you "fail" that test, you will be absorbed into the outgroup and lose sight of your old emotional/intuitive presuppositions and worldview. You will lose a lot of friends and sunk costs, and be subjected to new rules. The risk is to literally lose your old self. That's horrifying, that idea can feel something like being abducted by the borg from star trek and converted into one of them.

FeepingCreature 4y ago
Hence, keeping your identity small: http://www.paulgraham.com/identity.html http://www.paulgraham.com/identity.html
XorNot 4y ago
No, it isn't. Immersing yourself in someone else's propaganda just propagandizes you. The easiest victims are the ones who think they're genuinely too smart for it to have an effect on them. There's a reason country's pump this stuff out despite it seeming, on a cursory glance, so ridiculous and overt.

gwd 4y ago

> There's a danger, while steelmanning, that you might come up with an argument that sounds good to you, but it's actually weaker to the opposition. Therefore, while trying to steelman their argument, you actually fill it with holes. That's why normal practice is to check: "So I hear you saying something like this: <steelman argument>. Is that right?" > To truly do it, you have to embed yourself not just in their rational thought but also in their worldview. Indeed; the best word I have for being truly persuasive is "incarnational": You have to enter into the situation of the person you want to convince, and see the way out of the cave they're in from where they are now. > And I really mean the part about it being dangerous. In all honesty, we aren't the purely rational creatures we think we are. Honestly steelmanning an argument changes you. And maybe that's for the better in some situations, but I certainly wouldn't take that for granted. I dunno; I have more faith in the power of truth. You certainly need to be careful that you're not taking information only from one source; if you only listened to Goebel's propaganda for 2 years you'd certainly start to have Nazi patterns of thought infect your thinking. But if you want to rescue someone who has only heard Goebel's propaganda for the last several years, you have to learn how to recognize the flaws in thinking from within it.