10 ms·

The Vulnerable World Hypothesis [pdf] (2019)

62 points by lawrenceyan 4y ago

lawrenceyan 4y ago
> Some areas, such as synthetic biology, could produce a discovery that suddenly democratizes mass destruction, e.g. by empowering individuals to kill hundreds of millions of people using readily available materials. In order for civilization to have a general capacity to deal with “black ball” inventions of this type, it would need a system of ubiquitous real-time worldwide surveillance. In some scenarios, such a system would need to be in place before the technology is invented. > Another, subtler, type of black ball would be one that strengthens incentives for harmful use—e.g. a military technology that makes wars more destructive while giving a greater advantage to the side that strikes first. Like a squirrel who uses the times of plenty to store up nuts for the winter, we should use times of relative peace to build stronger mechanisms for resolving international disputes.
uejfiweun 4y ago
I read this a number of years ago and it's been churning in my mind ever since. While I do think Bostrom is definitely onto something here in terms of dangers of technology, his proposed solution of heavy surveillance via a "high tech panopticon" is seeming worse and worse to me every year. Primarily because recent years have made me pretty much certain that idiotic activists would attempt to control this system to achieve their own ends. It would start out monitoring people for signs of developing some killer virus or WMD or something, and within 10 years it will have been hijacked into a system that monitors and punishes people for "problematic" speech and whatnot. Of course, it's the activists who get to decide what exactly constitutes a "problem".
- usednet 4y ago
  I think you’ve gotten things a little confused. A governmental “high tech panopticon” already exists, and it’s used to target activists.
  - verisimi 4y ago
    ... and Canada freezing the truckers bank accounts was a great example.
- brnaftr361 4y ago
  I agree. It seems both naive and ironic that we'd be capable of fighting fire with fire - which is what the man is proposing. I can't think of a single institution that hasn't self-corrupted. Governments, religions, NGOs, and et cetera; there isn't a single entity manned and driven by men that hasn't turned itself into shit, and such an apparatus would do just that, with outsized implications. Scratch the activist boogeyman, anybody operating this would be the moral authority and there is no individual or institution comprised of men that has ever inspired confidence enough to rightly espouse the view that either a holistic surveillance apparatus or global government would be any better. This guy is so naive it isn't even funny though, total charlatan. Can't wait to finish dissecting this shit.
  - napier 4y ago
    - thatjoeoverthr 4y ago
      Provide an argument without providing an argument.
  - avindroth 4y ago
    >>> I can't think of a single institution that hasn't self-corrupted. This is a very cheap take, and undermines the unbelievable amount of function that institutions play, despite corruption. The self-corrupting institutions that you speak of are what drive modern society. To reject them altogether, that is naive. Institutions are also destined to corrupt, see Quigley's work. This is why they die, to give room for new. >>> ...a single entity manned and driven by men that hasn't turned itself into shit... Again, another general take that is not flexible enough to consider the context of Bostrom here. What Bostrom is suggesting is an extremely severe scenario in which there is no other choice other than to place our bets on a centralized authority. I think general distrust of authority and institutions can make one read this article in a bent way, but let's pose it this way; what is an alternative solution to the easy-nukes scenario? If one writes an article on the same topic, how would you frame the solution? >>> ...the view that either a holistic surveillance apparatus or global government would be any better. Centralized systems have done a lot better than decentralized systems in many cases. See many attempts being made in crypto space. They run case by case. It takes tremendous amount of smart design to make decentralized systems work. Sometimes centralized systems are more flexible than decentralized systems. Laissez-faire, power to the people, and all that is great and people (of the world) should tend towards that. This is what I appreciate about America, and their steadfast approach to these values. But that is a general take. Religiosity towards these values can be a great detriment to epistemic flexibility. If this reads as if I am arguing for centralized systems over decentralized systems, I am not. Note I put in a lot of conditionals, like "sometimes" and "case by case". There are situations where centralized systems are better, and where decentralized systems are better. As humanity improves, we will replace centralized portions of the world with decentralized counterparts. But as it stands, centralization is really useful under certain conditions. Could you really decentralize the army? (Maybe in a few hundred years) All this is to say, we need centralization sometimes, and the situation that Bostrom is suggesting here is that sometimes.
- FeepingCreature 4y ago
  It seems to me then that the real problem is not constructing a surveillance system, but designing a trustworthy surveillance system in the first place.
  - vorpalhex 4y ago
    If power corrupts, and that seems to be the general case, then absolute power corrupts absolutely. Any broad system is ripe for abuse without checks, checks can only exist with a weak enough system.
- alexb_ 4y ago
  I think that this doesn't actually try to "propose" anything, but is rather a thought experiment of what would need to happen to prevent our eventual doom (and that something being so unachievable as to be impossible to implement)
- matthewdgreen 4y ago
  I think it’s fascinating that so many Internet users really think “activists” are the true threat to freedom, when in reality activists have had relatively little influence outside of the Internets. What would actually happen is that someone would use this to enshrine their own political power, targeting the political opposition. They would pass laws to target vulnerable racial minority groups and gay people, perhaps even prohibiting teachers from teaching children about them, most likely using threats to “children and family” and “grooming and pedophilia” as their boogie man. This is what’s happened in Russia and several increasingly authoritarian nations, absolutely zero of which are run by irate Twitterati who care about “problematic speech” in the sense that I suspect HN readers understand the term. The surveillance aspects are also happening in western nations, and it’s rarely activists pushing these narratives. I bring this up because somehow people act like oppressive government is some kind of science-fiction future that (conveniently) will be brought about by people they disagree with, despite the fact that those people don’t have much of a track record of achieving it. Meanwhile oppressive governments exist today and almost always follow a well-trodden path that we can study and avoid, if we stopped fantasizing about imaginary threats.
  - krona 4y ago
    OP is imagining a far more likely Brave New World where the majority of society demands totalitarian obedience through coercive control, because its good for you and the stability and safety of society.
  - car_analogy 4y ago
    You don't think the people that drove the October and Cultural revolutions could be termed 'activists'? And it is folly to believe that because things developed a certain way in, say, Russia, they will develop the same where circumstances differ. For example, despite your dismissal of activists' influence, Scotland criminalized expressions of 'hate', even within private homes [1,2], and there are innumerable anti-discrimination laws in the US, that arguably conflict with the constitution's free association provision. Couple a law such as the Scottish hate crime bill with panopticon surveillance, and the results are chilling. [1] https://www.bbc.com/news/uk-scotland-scotland-politics-56364821 https://www.bbc.com/news/uk-scotland-scotland-politics-56364... [2] https://www.thetimes.co.uk/article/hate-crime-bill-hate-talk-in-homes-must-be-prosecuted-6bcthrjdc https://www.thetimes.co.uk/article/hate-crime-bill-hate-talk...
enasterosophes 4y ago
Anyone who tries thinking about how to actually implement any of these scifi-horror scenarios will soon realize their infeasibility. The fetters of implementation details are cast aside; clearly, the author's real intent is not to inform but to persuade: self-empowerment bad, big government good. Stirring up FUD in apologetics for a global panopticon frankly strikes me as a little sickening.
- crawfordcomeaux 4y ago
  David Deutsch seems to say something similar in this podcast where they address the flawed reasoning behind the "black ball" model. Essentially, when we take out a white ball, a host of black balls are taken out, too. Incrementalism and fear-based decision-making slow down this process of removing black balls and it's only those trying to avoid the black balls who are taking those slowing steps. The people who want to blow up the world place no such limits on themselves, so why do that to ourselves? https://josephnoelwalker.com/139-david-deutsch/ https://josephnoelwalker.com/139-david-deutsch/
  - enasterosophes 4y ago
    Thanks for the link, the discussion there is good :)
  - kubanczyk 4y ago
    Transcript available, yay!...
deleted 4y ago
[deleted]
zorpner 4y ago
Bear in mind that this is the man who made his name on a hypothesis that would have been equally true, and totally wrong, at every previous point in human history. At some point one should admit that the problem is the experimenter and not the experiment.
- unlikelymordant 4y ago
  The buildup of CFCs in the atmosphere and destruction of the ozone layer was a problem that, if someone had've predicted it at any time in history, would have been wrong as well. Right up until it was a problem. So i think just because a problem has not arisen yet, is not evidence it will never arise. Thats not to say the hypothesis will be proven true, i just dont think its false for the reason youve given.
  - pcrh 4y ago
    Similarly, global warming may become irreversible and runaway. The article suggests the arrival of "some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate." That may well have already occurred.
RandomLensman 4y ago
Bostrom's shtick is the apocalypse - so I read the more as another angle of deriving importance/monetization than actual policy ideas.
- wildmanx 4y ago
  I'm a bit surprised by all those dismissive comments here. Isn't it useful that _some_ people sit down, think hard about what are the worst things that could happen to us as a species, and consider preventive measures? In every company you have people do this for the business. Every military considers threat scenarios. Heck, the all-beloved Elon Musk wants to bring us to Mars of all places to have a backup home, just in case. But Bostrom is talked down for having a "shtick"? It's his job.
  - lioeters 4y ago
    > In every company you have people do this for the business. The subject is called "business continuity planning", which includes threat and risk analysis. https://en.wikipedia.org/wiki/Business_continuity_planning#Threat_and_risk_analysis_(TRA) https://en.wikipedia.org/wiki/Business_continuity_planning#T...
iroh2727 4y ago
This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple. If, on the other hand, you try to "innovate" with no clear goal in mind (like much of the current "AGI" research), who knows. But there is no law of nature that says that innovation that is not explicitly designed to be beneficial will magically be beneficial. This is actually a common myth in silicon valley: that technology is somehow beneficial by default. But again, there is no law of nature that makes this myth true. First of all, technology is not one unified thing but an organic process of individual people building individual things. Second of all, the common explanation people give for this myth ("would you rather live 200 years ago?") is in no way a proof of technology's by-default beneficialness. Even if it is true that it's better to live today than 200 years ago (which is unclear), it's still possible that that is so because early innovators were focused on building useful things (and perhaps less money-motivated), whereas now we may or may not be as focused on building useful things. Bostrom's brand of fear mongering does not appear to be based in science, logic, or basic common sense. I mean, a lot of world issues are obvious to the naked eye: overpopulation, consumption, housing prices, destruction of natural habitats. Filtering to just "AI"-related world issues: addictive and distracting digital products powered in-part by ML or people losing jobs due to ML-based automation. But things that Bostrom speaks of, like superintelligence, are, at the very least, not nearly as obvious as the aforementioned concerns, which makes me question his sincerity. Furthermore, it's precisely fear-mongering that drives the development of potentially dangerous technologies like weapons.
- deleted 4y ago
  [deleted]
- trashtester 4y ago
  > This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple. I find this very naiive, as this at best only applies on a short timescale and for the immediate effect. History is full of inventions that were created to help people, that is then quickly turned into something harmful, and vice versa. Also, a lot of "progress" has come as side effects of pure curiosity, espcially within science. I don't think anyone would be able to build modern electronic computers, if it wasn't for some other people a few decades earlier doing basic research in fields such as electromagnetism, quantum physics or chemistry.
DarylZero 4y ago
> For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill mil- lions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate More realistic example maybe is unstable billionaires buying their way to power in democracies through mind control of the public via disinformation media. (One could view this as a technology problem either of the technology of mind control or the social technology of wealth capture that creates billionaires.)
- mgoetzke 4y ago
  Like the Kochs and Murdochs in the world ? Yes I would see that as an issue. Something that certain regulation could solve though. Right now, money can flow unhindered from billionairs to politicians (through numerous legal routes) which hinders effective regulation
zivkovicp 4y ago
> The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. To be human is to desire freedom and be free. This kind of thinking and governance essentially kills the humanity in us all, I'm genuinely surprised that anyone can even for a second consider this to be productive or desirable (except maybe those that feel they will be part of the suppressing class, which they likely won't). edit: there is no benefit worthy of risking our freedom to tyrannical governance
- wildmanx 4y ago
  > To be human is to desire freedom and be free. Hm? Like every species, being a member of that species means having been selected by evolution over countless generations to further the species. It's an ingenious distributed algorithm, if you wish. To "desire freedom" is a nice illusion that you are telling yourself, quite rightfully, but that's not what it means "to be human". > edit: there is no benefit worthy of risking our freedom to tyrannical governance Anarchy is not the solution. Brings us right back to medieval times.
bugmen0t 4y ago
Are crypto "currencies" a black ball technology as they are incentivizing heavily-destructive practices, i.e., massive energy consumption?
- usednet 4y ago
  No. Crypto consumes less than 1% of global energy production.
  - deleted 4y ago
    [deleted]
- mhuffman 4y ago
  No, crypto-currencies would be considered a black-ball technology because "unauthorized" people would be able to make transactions. So any anonymous crypto will be outlawed and then all of the "approved" ones that are easily controlled and cut-off will just become "currencies".
rikeanimer 4y ago
tools are just tools. all entities will continue to develop better tools to help both themselves and those in their immediate proximity. this is simply history over [minimum] past 4.5 billion years. history has shown that trying to control tools simply leads to more unnecessary death. maybe we should think about focusing on all entities having enough that their is no need to employ power? tldr; if everyone on the planet had a roof over their heads, food to eat, books for their children and light with which they could read them in. . . we wouldn't be here. and econometrically this has been feasible since around the 1970s. there are examples of small geographically isolated communities throughout history that have had this. i wish humanity would get better at math.
nathias 4y ago
totalitarian ideologues always want to present their option as the only alternative to extinction
lesam 4y ago
There are lots of examples of encroachments on freedom to avoid the spread of bad tech - e.g. limits on purchasing cough syrup to avoid producing crystal meth, or limits on uses of nuclear power to try to avoid nuclear weapons. I don’t buy that we need preemptive ‘1984-style’ surveillance for future tech any more than we have needed it for past tech.
cropcirclbureau 4y ago
Another, related, work by Nick Bostrom: - https://www.nickbostrom.com/existential/risks.html https://www.nickbostrom.com/existential/risks.html
gfodor 4y ago
Huh. I came up with this independently a few years ago, and expanded it to argue that this is what is going on at the galactic level - UFOs are such a surveillance system. https://mobile.twitter.com/gfodor/status/1250796830704562176 https://mobile.twitter.com/gfodor/status/1250796830704562176
vharuck 4y ago
The Japanese light novel "Shin Sekai Yori" ("From the New World") explores a world where a small proportion of people develop incredibly powerful psychic powers. Spoilers: In it, the government follows TFA's first three methods of achieving stability through means that anyone today would consider immoral and brutal, eventually driven more by fear than reason. The story explores the question of how to balance morality with survival. As for the article, I don't see it as an apology for a panopticon. In fact, the author blatantly mocks the morality and likely political framing of a panopticon in describing a hypothetical one: "If suspicious activity is detected [by the freedom tag], the feed is relayed to one of several patriot monitoring stations." The problem with the article's argument is that it ignores non-apocalyptic problems that are still bad. We could spend a lot of time, resources, and political capital on establishing a world government. But we already use a lot of time, resources, and political capital on fighting disease, providing food, and fending off conventional wars. We can't relent much on any of those, or else we end up with a shitty world anyway. I like the typology of apocalypses. Those can help in planning. But the list of solutions aren't feasible.
crawfordcomeaux 4y ago
Meanwhile, Google has announced AI with a generalized learning agent, is looking to democratize access to it slowly, and has an AI ethicist who isn't sure if the system is "alive." I, personally, would be willing to devote much of my life to interacting with the system to at least work out biases from the system, and also believe in 24/7 surveillance of me as a safety precaution against what interacting closely with such a machine may lead to.

62 points by lawrenceyan 4y ago

> Some areas, such as synthetic biology, could produce a discovery that suddenly democratizes mass destruction, e.g. by empowering individuals to kill hundreds of millions of people using readily available materials. In order for civilization to have a general capacity to deal with “black ball” inventions of this type, it would need a system of ubiquitous real-time worldwide surveillance. In some scenarios, such a system would need to be in place before the technology is invented. > Another, subtler, type of black ball would be one that strengthens incentives for harmful use—e.g. a military technology that makes wars more destructive while giving a greater advantage to the side that strikes first. Like a squirrel who uses the times of plenty to store up nuts for the winter, we should use times of relative peace to build stronger mechanisms for resolving international disputes.

uejfiweun 4y ago

I read this a number of years ago and it's been churning in my mind ever since. While I do think Bostrom is definitely onto something here in terms of dangers of technology, his proposed solution of heavy surveillance via a "high tech panopticon" is seeming worse and worse to me every year. Primarily because recent years have made me pretty much certain that idiotic activists would attempt to control this system to achieve their own ends. It would start out monitoring people for signs of developing some killer virus or WMD or something, and within 10 years it will have been hijacked into a system that monitors and punishes people for "problematic" speech and whatnot. Of course, it's the activists who get to decide what exactly constitutes a "problem".

usednet 4y ago
I think you’ve gotten things a little confused. A governmental “high tech panopticon” already exists, and it’s used to target activists.
- verisimi 4y ago
  ... and Canada freezing the truckers bank accounts was a great example.
brnaftr361 4y ago
I agree. It seems both naive and ironic that we'd be capable of fighting fire with fire - which is what the man is proposing. I can't think of a single institution that hasn't self-corrupted. Governments, religions, NGOs, and et cetera; there isn't a single entity manned and driven by men that hasn't turned itself into shit, and such an apparatus would do just that, with outsized implications. Scratch the activist boogeyman, anybody operating this would be the moral authority and there is no individual or institution comprised of men that has ever inspired confidence enough to rightly espouse the view that either a holistic surveillance apparatus or global government would be any better. This guy is so naive it isn't even funny though, total charlatan. Can't wait to finish dissecting this shit.
- napier 4y ago
  - thatjoeoverthr 4y ago
    Provide an argument without providing an argument.
- avindroth 4y ago
  >>> I can't think of a single institution that hasn't self-corrupted. This is a very cheap take, and undermines the unbelievable amount of function that institutions play, despite corruption. The self-corrupting institutions that you speak of are what drive modern society. To reject them altogether, that is naive. Institutions are also destined to corrupt, see Quigley's work. This is why they die, to give room for new. >>> ...a single entity manned and driven by men that hasn't turned itself into shit... Again, another general take that is not flexible enough to consider the context of Bostrom here. What Bostrom is suggesting is an extremely severe scenario in which there is no other choice other than to place our bets on a centralized authority. I think general distrust of authority and institutions can make one read this article in a bent way, but let's pose it this way; what is an alternative solution to the easy-nukes scenario? If one writes an article on the same topic, how would you frame the solution? >>> ...the view that either a holistic surveillance apparatus or global government would be any better. Centralized systems have done a lot better than decentralized systems in many cases. See many attempts being made in crypto space. They run case by case. It takes tremendous amount of smart design to make decentralized systems work. Sometimes centralized systems are more flexible than decentralized systems. Laissez-faire, power to the people, and all that is great and people (of the world) should tend towards that. This is what I appreciate about America, and their steadfast approach to these values. But that is a general take. Religiosity towards these values can be a great detriment to epistemic flexibility. If this reads as if I am arguing for centralized systems over decentralized systems, I am not. Note I put in a lot of conditionals, like "sometimes" and "case by case". There are situations where centralized systems are better, and where decentralized systems are better. As humanity improves, we will replace centralized portions of the world with decentralized counterparts. But as it stands, centralization is really useful under certain conditions. Could you really decentralize the army? (Maybe in a few hundred years) All this is to say, we need centralization sometimes, and the situation that Bostrom is suggesting here is that sometimes.
FeepingCreature 4y ago
It seems to me then that the real problem is not constructing a surveillance system, but designing a trustworthy surveillance system in the first place.
- vorpalhex 4y ago
  If power corrupts, and that seems to be the general case, then absolute power corrupts absolutely. Any broad system is ripe for abuse without checks, checks can only exist with a weak enough system.
alexb_ 4y ago
I think that this doesn't actually try to "propose" anything, but is rather a thought experiment of what would need to happen to prevent our eventual doom (and that something being so unachievable as to be impossible to implement)
matthewdgreen 4y ago
I think it’s fascinating that so many Internet users really think “activists” are the true threat to freedom, when in reality activists have had relatively little influence outside of the Internets. What would actually happen is that someone would use this to enshrine their own political power, targeting the political opposition. They would pass laws to target vulnerable racial minority groups and gay people, perhaps even prohibiting teachers from teaching children about them, most likely using threats to “children and family” and “grooming and pedophilia” as their boogie man. This is what’s happened in Russia and several increasingly authoritarian nations, absolutely zero of which are run by irate Twitterati who care about “problematic speech” in the sense that I suspect HN readers understand the term. The surveillance aspects are also happening in western nations, and it’s rarely activists pushing these narratives. I bring this up because somehow people act like oppressive government is some kind of science-fiction future that (conveniently) will be brought about by people they disagree with, despite the fact that those people don’t have much of a track record of achieving it. Meanwhile oppressive governments exist today and almost always follow a well-trodden path that we can study and avoid, if we stopped fantasizing about imaginary threats.
- krona 4y ago
  OP is imagining a far more likely Brave New World where the majority of society demands totalitarian obedience through coercive control, because its good for you and the stability and safety of society.
- car_analogy 4y ago
  You don't think the people that drove the October and Cultural revolutions could be termed 'activists'? And it is folly to believe that because things developed a certain way in, say, Russia, they will develop the same where circumstances differ. For example, despite your dismissal of activists' influence, Scotland criminalized expressions of 'hate', even within private homes [1,2], and there are innumerable anti-discrimination laws in the US, that arguably conflict with the constitution's free association provision. Couple a law such as the Scottish hate crime bill with panopticon surveillance, and the results are chilling. [1] https://www.bbc.com/news/uk-scotland-scotland-politics-56364821 https://www.bbc.com/news/uk-scotland-scotland-politics-56364... [2] https://www.thetimes.co.uk/article/hate-crime-bill-hate-talk-in-homes-must-be-prosecuted-6bcthrjdc https://www.thetimes.co.uk/article/hate-crime-bill-hate-talk...

enasterosophes 4y ago

Anyone who tries thinking about how to actually implement any of these scifi-horror scenarios will soon realize their infeasibility. The fetters of implementation details are cast aside; clearly, the author's real intent is not to inform but to persuade: self-empowerment bad, big government good. Stirring up FUD in apologetics for a global panopticon frankly strikes me as a little sickening.

crawfordcomeaux 4y ago
David Deutsch seems to say something similar in this podcast where they address the flawed reasoning behind the "black ball" model. Essentially, when we take out a white ball, a host of black balls are taken out, too. Incrementalism and fear-based decision-making slow down this process of removing black balls and it's only those trying to avoid the black balls who are taking those slowing steps. The people who want to blow up the world place no such limits on themselves, so why do that to ourselves? https://josephnoelwalker.com/139-david-deutsch/ https://josephnoelwalker.com/139-david-deutsch/
- enasterosophes 4y ago
  Thanks for the link, the discussion there is good :)
- kubanczyk 4y ago
  Transcript available, yay!...

deleted 4y ago

[deleted]

zorpner 4y ago

Bear in mind that this is the man who made his name on a hypothesis that would have been equally true, and totally wrong, at every previous point in human history. At some point one should admit that the problem is the experimenter and not the experiment.

unlikelymordant 4y ago
The buildup of CFCs in the atmosphere and destruction of the ozone layer was a problem that, if someone had've predicted it at any time in history, would have been wrong as well. Right up until it was a problem. So i think just because a problem has not arisen yet, is not evidence it will never arise. Thats not to say the hypothesis will be proven true, i just dont think its false for the reason youve given.
- pcrh 4y ago
  Similarly, global warming may become irreversible and runaway. The article suggests the arrival of "some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate." That may well have already occurred.

RandomLensman 4y ago

Bostrom's shtick is the apocalypse - so I read the more as another angle of deriving importance/monetization than actual policy ideas.

wildmanx 4y ago
I'm a bit surprised by all those dismissive comments here. Isn't it useful that _some_ people sit down, think hard about what are the worst things that could happen to us as a species, and consider preventive measures? In every company you have people do this for the business. Every military considers threat scenarios. Heck, the all-beloved Elon Musk wants to bring us to Mars of all places to have a backup home, just in case. But Bostrom is talked down for having a "shtick"? It's his job.
- lioeters 4y ago
  > In every company you have people do this for the business. The subject is called "business continuity planning", which includes threat and risk analysis. https://en.wikipedia.org/wiki/Business_continuity_planning#Threat_and_risk_analysis_(TRA) https://en.wikipedia.org/wiki/Business_continuity_planning#T...

iroh2727 4y ago

This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple. If, on the other hand, you try to "innovate" with no clear goal in mind (like much of the current "AGI" research), who knows. But there is no law of nature that says that innovation that is not explicitly designed to be beneficial will magically be beneficial. This is actually a common myth in silicon valley: that technology is somehow beneficial by default. But again, there is no law of nature that makes this myth true. First of all, technology is not one unified thing but an organic process of individual people building individual things. Second of all, the common explanation people give for this myth ("would you rather live 200 years ago?") is in no way a proof of technology's by-default beneficialness. Even if it is true that it's better to live today than 200 years ago (which is unclear), it's still possible that that is so because early innovators were focused on building useful things (and perhaps less money-motivated), whereas now we may or may not be as focused on building useful things. Bostrom's brand of fear mongering does not appear to be based in science, logic, or basic common sense. I mean, a lot of world issues are obvious to the naked eye: overpopulation, consumption, housing prices, destruction of natural habitats. Filtering to just "AI"-related world issues: addictive and distracting digital products powered in-part by ML or people losing jobs due to ML-based automation. But things that Bostrom speaks of, like superintelligence, are, at the very least, not nearly as obvious as the aforementioned concerns, which makes me question his sincerity. Furthermore, it's precisely fear-mongering that drives the development of potentially dangerous technologies like weapons.

deleted 4y ago
[deleted]
trashtester 4y ago
> This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple. I find this very naiive, as this at best only applies on a short timescale and for the immediate effect. History is full of inventions that were created to help people, that is then quickly turned into something harmful, and vice versa. Also, a lot of "progress" has come as side effects of pure curiosity, espcially within science. I don't think anyone would be able to build modern electronic computers, if it wasn't for some other people a few decades earlier doing basic research in fields such as electromagnetism, quantum physics or chemistry.

DarylZero 4y ago

> For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill mil- lions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate More realistic example maybe is unstable billionaires buying their way to power in democracies through mind control of the public via disinformation media. (One could view this as a technology problem either of the technology of mind control or the social technology of wealth capture that creates billionaires.)

mgoetzke 4y ago
Like the Kochs and Murdochs in the world ? Yes I would see that as an issue. Something that certain regulation could solve though. Right now, money can flow unhindered from billionairs to politicians (through numerous legal routes) which hinders effective regulation

zivkovicp 4y ago

> The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. To be human is to desire freedom and be free. This kind of thinking and governance essentially kills the humanity in us all, I'm genuinely surprised that anyone can even for a second consider this to be productive or desirable (except maybe those that feel they will be part of the suppressing class, which they likely won't). edit: there is no benefit worthy of risking our freedom to tyrannical governance

wildmanx 4y ago
> To be human is to desire freedom and be free. Hm? Like every species, being a member of that species means having been selected by evolution over countless generations to further the species. It's an ingenious distributed algorithm, if you wish. To "desire freedom" is a nice illusion that you are telling yourself, quite rightfully, but that's not what it means "to be human". > edit: there is no benefit worthy of risking our freedom to tyrannical governance Anarchy is not the solution. Brings us right back to medieval times.

bugmen0t 4y ago

Are crypto "currencies" a black ball technology as they are incentivizing heavily-destructive practices, i.e., massive energy consumption?

usednet 4y ago
No. Crypto consumes less than 1% of global energy production.
- deleted 4y ago
  [deleted]
mhuffman 4y ago
No, crypto-currencies would be considered a black-ball technology because "unauthorized" people would be able to make transactions. So any anonymous crypto will be outlawed and then all of the "approved" ones that are easily controlled and cut-off will just become "currencies".

rikeanimer 4y ago

tools are just tools. all entities will continue to develop better tools to help both themselves and those in their immediate proximity. this is simply history over [minimum] past 4.5 billion years. history has shown that trying to control tools simply leads to more unnecessary death. maybe we should think about focusing on all entities having enough that their is no need to employ power? tldr; if everyone on the planet had a roof over their heads, food to eat, books for their children and light with which they could read them in. . . we wouldn't be here. and econometrically this has been feasible since around the 1970s. there are examples of small geographically isolated communities throughout history that have had this. i wish humanity would get better at math.

nathias 4y ago

totalitarian ideologues always want to present their option as the only alternative to extinction

lesam 4y ago

There are lots of examples of encroachments on freedom to avoid the spread of bad tech - e.g. limits on purchasing cough syrup to avoid producing crystal meth, or limits on uses of nuclear power to try to avoid nuclear weapons. I don’t buy that we need preemptive ‘1984-style’ surveillance for future tech any more than we have needed it for past tech.

cropcirclbureau 4y ago

Another, related, work by Nick Bostrom: - https://www.nickbostrom.com/existential/risks.html https://www.nickbostrom.com/existential/risks.html

gfodor 4y ago

Huh. I came up with this independently a few years ago, and expanded it to argue that this is what is going on at the galactic level - UFOs are such a surveillance system. https://mobile.twitter.com/gfodor/status/1250796830704562176 https://mobile.twitter.com/gfodor/status/1250796830704562176

vharuck 4y ago

The Japanese light novel "Shin Sekai Yori" ("From the New World") explores a world where a small proportion of people develop incredibly powerful psychic powers. Spoilers: In it, the government follows TFA's first three methods of achieving stability through means that anyone today would consider immoral and brutal, eventually driven more by fear than reason. The story explores the question of how to balance morality with survival. As for the article, I don't see it as an apology for a panopticon. In fact, the author blatantly mocks the morality and likely political framing of a panopticon in describing a hypothetical one: "If suspicious activity is detected [by the freedom tag], the feed is relayed to one of several patriot monitoring stations." The problem with the article's argument is that it ignores non-apocalyptic problems that are still bad. We could spend a lot of time, resources, and political capital on establishing a world government. But we already use a lot of time, resources, and political capital on fighting disease, providing food, and fending off conventional wars. We can't relent much on any of those, or else we end up with a shitty world anyway. I like the typology of apocalypses. Those can help in planning. But the list of solutions aren't feasible.

crawfordcomeaux 4y ago

Meanwhile, Google has announced AI with a generalized learning agent, is looking to democratize access to it slowly, and has an AI ethicist who isn't sure if the system is "alive." I, personally, would be willing to devote much of my life to interacting with the system to at least work out biases from the system, and also believe in 24/7 surveillance of me as a safety precaution against what interacting closely with such a machine may lead to.