3 ms·

Hiya Chubot, (fan of Oil by the way, thanks so much for your work!) I read the article you linked to, both parts. I wonder how much of people having psychotic

Hiya Chubot, (fan of Oil by the way, thanks so much for your work!) I read the article you linked to, both parts. I wonder how much of people having psychotic breaks in the rationalist community is due to 1) people with tendencies toward mental illness gravitating toward rationalism or 2) rationalism being a way to avoid built-in biases in human thought, but those biases being important to keeping us sane on an individual level. (If you fully grasp the idea that everyone might die, and have an emotional reaction to that that's proportional compared to just one person you know, it can be devastating). I think we are bad at thinking about big numbers and risks, because being very good at evaluating risks is actually not great for short term, individual survival -- even if it's good for survival as a species. I know personally the whole AI/AGI thing has got me really down. It's a struggle to reconcile it with how little a lot of people seem to put stock in the idea of AGI ending up in control of humanity at some point. I totally agree that everything on your list is a real issue -- but I think that, even if we completely solve all those issues, how do we not end up with a society where most important decisions are eventually made by AGI? My assumptions are 1) that we eventually make AGI which is just superior to humans in terms of making decisions and planning, and 2) there will be significant pressure from capitalism and competition among governments to use AGI over people once that's the case. Similar to how automation has almost always won out over hand-production so far for manufactured goods. That's more the scenario Paul Christiano worries about than Yudkowsky. It seems more likely to me. But I still think that a lot of our mental heuristics about what we should worry about break down when it comes to the creation of something that out-guns us in terms of brainpower, and I think Yudkowsky makes a lot of good points about how we tend to shy away from facing reality when reality could be dreadful. That it's really easy to have a mental block about inventing something that makes humanity go extinct, where if it's possible to do that, there's no outside force that will swoop in and stop us like a parent stopping a child from falling into a river. If this is a real danger, we have to identify it in advance and take action to prevent it, even while there are a bunch of other problems to deal with, and even while a lot of people don't believe in it, and even while there's a lot of money to be made in the meantime but each bit takes us closer to a bad outcome. Reality isn't necessarily fair, we could be really screwed, and have all the problems you mentioned in addition to the risk of AGI killing us all (either right away or taking over and just gradually using up all the resources we need to live like we've done to so many species).

chubot 3y ago
There were 7 parts, and the last one actually addresses exactly the issue we're talking about: https://aiascendant.substack.com/p/extropias-children-chapter-7 https://aiascendant.substack.com/p/extropias-children-chapte... (and I still recommend all 7 parts) It's hard for me to respond to the rest your comments, because I simply don't agree with the framing. To me I see a big trap where people read a lot of words, but don't connect those words to reality, which requires action. If you only manipulate words (e.g. "intelligence" and "AGI" are big ones), then you're prone to believing in and reflecting illusions. He was upset that I wasn't upset. --- You didn't ask for advice, but you say you're not feeling good. I would draw an analogy to a friend who in 2017 was preoccupied with Trump possibly causing a nuclear war with North Korea. There were a lot of memes in the news about this. Now certainly I couldn't say at the time that it was impossible. But looking back 6 years, I'm glad that I completely ignored it and just lived my life. And that isn't to say it's gone, and not real -- certainly something like that could happen in the future, given the volatile political situation. But it's simply out of my control. (And that's what I told my friend -- are you going to DO anything about it?) Regardless of what you believe, I think if you just stop reading memes, and do other stuff instead, you won't regret it in 6 years. I agree with the Substack in that rationalists can be curiously irrational when it comes to examining where they get their own beliefs. (Or maybe examining them in a "logical" way, but completely missing the obvious, common-sense fact -- like MacAskill writing an entire book about moral actions, while his life's work was funded by a criminal -- a close associate who he vouched for!) Like EVERYONE ELSE, they are getting beliefs from their reptile brain ("world will definitely end"), and then they are CONFABULATING logical arguments on top to justify what the reptile brain wants. And you can't really change your reptile brain's mind by "thinking about it" -- it responds more to actions and experiences than the neocortex. It definitely responds to social status and having friends, which can lead to cult-like behavior. I'd say that humans have a "circuit" where they tend to believe the "average" of what people around them believe. So if you read a lot of "rationalist" stuff, you're going to believe it, even if's not grounded in experience, and there are gaping holes in the logic. --- The big trick I see a lot of "rationalist" writing is the same one used in the Bostrom book. 1. You state your premises in a page or so. Most of the premises are reasonable, but maybe one is vague or slightly wrong. You admit doubt. You come across as reasonable. 2. You spend 300 pages extrapolating wildly from those premises, creating entire worlds. You invent really fun ideas, like talking about a trilling trillion people, a trillion years into the future, across galaxies, and what they want. You use one of those slightly vague premises, or maybe a play on a word like "intelligence", to make these extrapolations seem like logical deductions. You overthink it. Other people who like overthinking like what you wrote. You lack the sense of proportion that people who have to act must acquire -- people with skin the game. Following this methodology is a good way to lead to self-contained thoughts, divorced from reality. The thoughts feed on themselves both individually and socially. If you're wrong, and you frequently are, you can just draw attention to some new fun ideas. --- Anyway, glad you have followed the project. Speaking from experience, I would suggest concretely to replace the rumination time with work on an open source project. If not coding, it could be testing or documentation, etc. Having a sense of achievement may make past negative thoughts of things far away seem silly.

chubot 3y ago

There were 7 parts, and the last one actually addresses exactly the issue we're talking about: https://aiascendant.substack.com/p/extropias-children-chapter-7 https://aiascendant.substack.com/p/extropias-children-chapte... (and I still recommend all 7 parts) It's hard for me to respond to the rest your comments, because I simply don't agree with the framing. To me I see a big trap where people read a lot of words, but don't connect those words to reality, which requires action. If you only manipulate words (e.g. "intelligence" and "AGI" are big ones), then you're prone to believing in and reflecting illusions. He was upset that I wasn't upset. --- You didn't ask for advice, but you say you're not feeling good. I would draw an analogy to a friend who in 2017 was preoccupied with Trump possibly causing a nuclear war with North Korea. There were a lot of memes in the news about this. Now certainly I couldn't say at the time that it was impossible. But looking back 6 years, I'm glad that I completely ignored it and just lived my life. And that isn't to say it's gone, and not real -- certainly something like that could happen in the future, given the volatile political situation. But it's simply out of my control. (And that's what I told my friend -- are you going to DO anything about it?) Regardless of what you believe, I think if you just stop reading memes, and do other stuff instead, you won't regret it in 6 years. I agree with the Substack in that rationalists can be curiously irrational when it comes to examining where they get their own beliefs. (Or maybe examining them in a "logical" way, but completely missing the obvious, common-sense fact -- like MacAskill writing an entire book about moral actions, while his life's work was funded by a criminal -- a close associate who he vouched for!) Like EVERYONE ELSE, they are getting beliefs from their reptile brain ("world will definitely end"), and then they are CONFABULATING logical arguments on top to justify what the reptile brain wants. And you can't really change your reptile brain's mind by "thinking about it" -- it responds more to actions and experiences than the neocortex. It definitely responds to social status and having friends, which can lead to cult-like behavior. I'd say that humans have a "circuit" where they tend to believe the "average" of what people around them believe. So if you read a lot of "rationalist" stuff, you're going to believe it, even if's not grounded in experience, and there are gaping holes in the logic. --- The big trick I see a lot of "rationalist" writing is the same one used in the Bostrom book. 1. You state your premises in a page or so. Most of the premises are reasonable, but maybe one is vague or slightly wrong. You admit doubt. You come across as reasonable. 2. You spend 300 pages extrapolating wildly from those premises, creating entire worlds. You invent really fun ideas, like talking about a trilling trillion people, a trillion years into the future, across galaxies, and what they want. You use one of those slightly vague premises, or maybe a play on a word like "intelligence", to make these extrapolations seem like logical deductions. You overthink it. Other people who like overthinking like what you wrote. You lack the sense of proportion that people who have to act must acquire -- people with skin the game. Following this methodology is a good way to lead to self-contained thoughts, divorced from reality. The thoughts feed on themselves both individually and socially. If you're wrong, and you frequently are, you can just draw attention to some new fun ideas. --- Anyway, glad you have followed the project. Speaking from experience, I would suggest concretely to replace the rumination time with work on an open source project. If not coding, it could be testing or documentation, etc. Having a sense of achievement may make past negative thoughts of things far away seem silly.