10 ms·

Dude, You Broke the Future (2017)

243 points by thanhhaimai 6y ago

pronoiac 6y ago
(2017)
coldtea 6y ago
Not so sure about his future prediction ability. This, for example, like most "the sky is falling", didn't age well: https://www.antipope.org/charlie/blog-static/2018/07/that-sinking-feeling.html https://www.antipope.org/charlie/blog-static/2018/07/that-si...
- nikanj 6y ago
  Brexit hasn't happened yet. The transition period ends in about six months, so it's too early to say if he was right or wrong
- klenwell 6y ago
  It looks to me like most Stross's predictions depend first on the event of the Brexit, which has not occurred yet as far as I know. Here's a map the NY Times published on 6/15/2005: http://graphics8.nytimes.com/images/2005/06/15/business/arm3.gif http://graphics8.nytimes.com/images/2005/06/15/business/arm3... It makes an implicit prediction about the then booming housing market and the sky above it. On that date, the S&P 500 closed at 1,206.58. Two years later, in 2007, the S&P 500 closed at 1,522.97. To the extent that stock markets tell us where we stand as a nation or a society, I find this example instructive. There's such a thing as the long view. It's hard to know how long it should be. Harder to know whether it will be proved out in the end until it is proved or disproved. Even then sometimes we miss the forest for the trees. S&P Data: https://finance.yahoo.com/quote/%5EGSPC/history?period1=1118793600&period2=1181865600&interval=1d&filter=history&frequency=1d https://finance.yahoo.com/quote/%5EGSPC/history?period1=1118...
- rsynnott 6y ago
  Remember that Brexit hasn’t, for practical purposes, happened yet. Most people didn’t anticipate that the UK would request, and be granted, so many extensions, a few years ago. However, time has now run out; the last realistic opportunity for a further extension passed a couple of weeks ago, and Brexit will now happen in some real sense at the end of the year. It likely won’t be pretty.
deleted 6y ago
[deleted]
dang 6y ago
If curious see also 2018 https://news.ycombinator.com/item?id=16051337 https://news.ycombinator.com/item?id=16051337 2017 https://news.ycombinator.com/item?id=16032643 https://news.ycombinator.com/item?id=16032643
jengleton 6y ago
Robert Miles has an interesting response to the claim that corporations are AIs: https://www.youtube.com/watch?v=L5pUA3LsEaw https://www.youtube.com/watch?v=L5pUA3LsEaw To paraphrase: yes, corporations function as agents, but their maximum performance is limited by the capabilities of their employees. A corporate AI may have a much broader set of skills than any single person, and may be able to tackle many concurrent tasks, but as an "intelligent agent" its decision making capabilities probably don't scale exponentially (or even linearly) with its headcount. In the sense that a corporation's maximum intelligence is likely to be in the same ballpark as the smartest humans, it can't be seen as a true superintelligence.
- zaphar 6y ago
  I don't think the comparison was to AI as a super intelligence. AI doesn't have to be a super intelligence to be AI or to be dangerous.
  - WrtCdEvrydy 6y ago
    I think AS (Artificial Stupidity) will get us first.
    - nekopa 6y ago
      Just NO. Artificial Stupidity (AS) will never be able to perform at the level of Actually Seen Stupidity (ASS). No matter how much you try to improve AS, God will always create a better ASS. You will always be limited by priors to create an AS, but ASS will evolve naturally, plus if God ever decides to use science and apply a Directed Acyclic Tangent (DAT) well, DAT ASS would be unbeatable.
  - jengleton 6y ago
    That's fair. I interpreted the comparison as being between "AI implemented as a property of human organizations" and "AI implemented as a powerful search algorithm". While corporations can certainly be dangerous, they're still made of people and thus are unlikely to want to do things like "convert the entire mass of the solar system into dollar bills" (and even if they did, they'd have a hard time doing it). A sufficiently powerful search algorithm would find all sorts of bizarre ways of satisfying its goals. The point is that the scope of the risk is quite different when talking about corporations (order of magnitude: screwing up the environment in pursuit of easy profit) vs "real" AI which, if granted agency, would have potentially unbounded risk. Another way to look at it is with regards to capacity for self-modification. If a corporation can't re-structure itself into being much smarter than the smartest human, its intelligence is fundamentally limited, and therefore so is the risk. Does software have this restriction? We don't really know yet, but it's hard to point to exactly why it would.
anchpop 6y ago
> And once you start probing the nether regions of transhumanist thought and run into concepts like Roko's Basilisk—by the way, any of you who didn't know about the Basilisk before are now doomed to an eternity in AI hell—you realize they've mangled it to match some of the nastiest ideas in Presybterian Protestantism. I wish every discussion of transhumanism didn't have to involve Roko's Basilisk. It's not something anyone takes seriously (and very few ever did), but it has enough quirky weirdness that everyone seems to want to talk about it. Here's a quote from "Lesswrongwiki": > Roko's argument was broadly rejected on Less Wrong, with commenters objecting that an agent like the one Roko was describing would have no real reason to follow through on its threat > [...] > Less Wrong's founder, Eliezer Yudkowsky, banned discussion of Roko's basilisk on the blog for several years as part of a general site policy against spreading potential information hazards. This had the opposite of its intended effect: a number of outside websites began sharing information about Roko's basilisk, as the ban attracted attention to this taboo topic. Websites like RationalWiki spread the assumption that Roko's basilisk had been banned because Less Wrong users accepted the argument; thus many criticisms of Less Wrong cite Roko's basilisk as evidence that the site's users have unconventional and wrong-headed beliefs. - https://wiki.lesswrong.com/wiki/Roko's_basilisk https://wiki.lesswrong.com/wiki/Roko's_basilisk
- scotty79 6y ago
  For me this illustrates nicely why most discussions about AI and singularity are like counting how many angels can dance on the head of a pin. It's all like talking about nuclear weapons in 17th century. Sure you can but even if you reach some conclusions they most likely will be invalidated by first technical detail of the implementation of thing when it actually gets created.
- LiquidSky 6y ago
  Your comment (and your links) miss the point of mocking references to "Roko's Basilisk". Whether it's actually a widely-held belief of the Less Wrong/greater rationalist community is irrelevant, it's emblematic of the eye-rolling nonsense that they engage in. It's the exact secular equivalent of "How many angels can dance on the head of a pin?"[1], which was also not an actual accepted topic of medieval religious scholarly debate but an illustration of the usual absurdity of such debate (e.g., there was an actual centuries-long scholarly discussion of whether angels were sexless or had sexes). [1]https://en.wikipedia.org/wiki/How_many_angels_can_dance_on_the_head_of_a_pin%3F https://en.wikipedia.org/wiki/How_many_angels_can_dance_on_t...
autokad 6y ago
>"Science fiction is written by people embedded within a society with expectations and political assumptions that bias us towards looking at the shiny surface of new technologies rather than asking how human beings will use them, and to taking narratives of progress at face value rather than asking what hidden agenda they serve." And be prepared for the most agenda ridden text you have read in a long time.
- claudiawerner 6y ago
  That's not necessarily a bad thing; in fact, most works (and specifically blogs) online have an "agenda".
  - Nasrudith 6y ago
    Not having any agenda is a deeply unhealthy mental state as it implies a complete apathy to all outcomes. It is just that most "normal" agendas are effectively invisible as a Huffman encoded to a 0 bit default. Not wanting to be tortured to death is an agenda technically but contributes so little information that it only makes sense to state the opposite.
- thiagocsf 6y ago
  Would you mind explaining what the agenda is?
- danielscrubs 6y ago
  He was also all over the place. But interesting ideas and if he can connect the ideas in a better way in his book I wouldn’t mind reading it.
- loa_in_ 6y ago
  That's absolutely not what I associate science fiction with. Stanisław Lem for example couldn't be further from taking shiny technology at face value
  - aasasd 6y ago
bawolff 6y ago
> Nobody in 2007 was expecting a Nazi revival in 2017, right? Is that actually true? I seem to remember people predicting that usa has been edging towards facism for a while now, especially in the aftermath of 9/11. For example here is an article from 2003 predicting usa is heading towards fasicism https://secularhumanism.org/2003/03/fascism-anyone/ https://secularhumanism.org/2003/03/fascism-anyone/
- bitwize 6y ago
  Nobody among the intellectual class anticipated Trump's election in 2016. Much like physicists hypothesized dark matter to account for the apparent greater mass necessary to stabilize galaxies and such, the political intelligentsia hypothesized an enormous, hidden body of "dark Nazis" to account for their chosen candidate not winning the election. It is unfathomable to them that enough people in the center, center-right, and right could be discontent enough with being told whom to vote for and why someone running on a platform of tighter immigration controls and more jobs for Americans is evil incarnate, that they were willing to "hold their nose and vote for Trump", to swing the election in his favor. Political intellectuals tend to lean left, and whenever a rightist scores a major political victory they start predicting stormtroopers goosestepping through American streets Real Soon Now, going back to at least Reagan. So the Nazi Revival is generally accepted as real, and if you contest the idea's truth you may be considered one of them.
  - jrumbut 6y ago
    This is very untrue, even if we take nobody as a little overstatement, half of the election prediction models had him above 10% chance of winning (FiveThirtyEight had him at 29%, https://www.nytimes.com/interactive/2016/upshot/presidential-polls-forecast.html?_r=0#other-forecasts https://www.nytimes.com/interactive/2016/upshot/presidential...). WBUR, Boston's NPR news station, published in their blog "Cognoscenti" an article titled "Why Donald Trump will Win in November" in May of 2016 (https://www.wbur.org/cognoscenti/2016/05/06/election2016-trump-establishment-politics-tom-keane https://www.wbur.org/cognoscenti/2016/05/06/election2016-tru...). I can't think of anything more representative of what people think of as the left leaning intellectual class than a Boston NPR blog with a Latin title. This idea that him winning in 2016 was inconceivable to media/intellectual circles is a bit of revisionist history.
Ygg2 6y ago
> nastiest ideas in Presybterian Protestantism. And if you follow Charlie's reasoning, that simpler parts, can't lead to a highly advanced intelligence, you come to conclusion intelligence is derived from a divine soul. Intelligence isn't as uncommon as we think, but it takes long time to emerge.
- hwillis 6y ago
  > if you follow Charlie's reasoning, that simpler parts, can't lead to a highly advanced intelligence, you come to conclusion intelligence is derived from a divine soul. Thats an absolutely ridiculous interpretation. Asserting that our current "simple parts" are still very far from AGI is nowhere even close to asserting that intelligence is immaterial. Its very, very obvious that there are myriad possible leaps we might need to get closer to intelligence. You can't build an attention network from a perceptron. Its totally reasonable to say that we need a few more fundamental discoveries first. Obvious, even, given the insane amount of processing required to teach a network a language. Brains are clearly wired in a way that is simply more efficient than how we currently know how to build stuff.
  - Ygg2 6y ago
    > Thats an absolutely ridiculous interpretation. Asserting that our current "simple parts" are still very far from AGI is nowhere even close to asserting that intelligence is immaterial. Not really what it says. Look at following passage: > AI singularity as a narrative, and identify the numerous places in the story where the phrase "... and then a miracle happens" occurs, it becomes apparent pretty quickly that they've reinvented Christianity. The phrase "miracle happens" , to me is suspect. Intelligence rose many times in various creatures. There is nothing miracle about something that rose multiple number of times. Are Singularists wrong? Yes. They confuse saturation for exponential curves; current neural networks are far cry from actual neuron networks, and their time scales reflect more their fear of death than any sensible timeline. If Charlie wants to criticize Singularists, there are plenty of valid reasons. Them being cult like is the least important one.
    - 6y ago
ALittleLight 6y ago
I found a couple errors early on that dissuaded me from continuing to read. One was the idea that the author could rule out the singularity because he wasn't aware of progress toward self-motivated AI that would be something like our own intelligence. This seems like a limited view to me because we wouldn't need self-motivation at all in AI to hit the singularity. Humans can supply the self motivation. Suppose I'm using GPT-4 or GPT-44 trained on the corpus from sci-hub and it recommends experiments to me, or explains physics to me, etc. I could be the self-motivating part and the AI could be the intelligence part, and it seems we'd still hit the technological singularity. Another problem I had was when the author characterized Elon Musk's "obsession" with the paperclip maximizer and described Tesla as a battery maximizer. It seems like the author kind of misses the point of the paperclip thought experiment, which is, broadly, that an AI's interests might not be aligned with our own and that misalignment may cause serious problems. Tesla is clearly not a battery maximizer and it is clearly not a different class of intelligence from the humans and corporations existing today (though it may be towards the top of that class). Neither of those things would necessarily be true of an AI. Given the position of my scrollbar it seems I was only starting to read this piece, but already finding what I think are significant problems as the author sets up the argument, I'm hesitant to spend more reading.
- worldsayshi 6y ago
  > Tesla [...] is clearly not a different class of intelligence from [...] corporations I believe he just used Tesla as an example of corporation not as something new or special. > the point of the paperclip thought experiment is, broadly, that an AI's interests might not be aligned with our own and that misalignment may cause serious problems. But (some) corporations clearly do share this aspect to some extent. They maximize profit while environmental concerns are not prioritized. And he pointed out later that corporations are slightly more complex than just maximizing 'one' thing. But a range of things and most importantly profit.
  - ALittleLight 6y ago
    Most, if not all, corporations are misaligned to humanity's well-being. That is, they want money, and that's not exactly the same thing as what would be best for humanity. This is okay though because corporations are the same class and kind of intelligence as the rest of humanity and so they can be (somewhat) predicted and constrained by our laws and norms. Because corporations are the same class of intelligence, they can't outsmart everyone. We can catch them when they do wrong and punish them and use laws and courts to control them. Because corporations are the same kind of intelligence they (usually) aren't going to do things we might consider insane or sociopathic. For example, if corporation X realized they might make a profit by taking out life insurance on workers and then getting them to do very risky things, X Corp probably wouldn't do it and if they did people would leak it and laws would be made to stop it. The paperclip thought experiment is an illustration of a different kind of intelligence that wouldn't be constrained by human morality or norms and would surpass human intelligence and not be constrained by laws or force either.
jariel 6y ago
Corporations are nothing but a group of actors working together to achieve some outcome, part of which may be profit, and it's absurd to think that they are a 'Modern Era', or even Western concept as every single land-owning farmer, artisan, banker, merchant in history was de-facto a form of 'corporation'. Moreover, many institutions with origins in antiquity (schools, governments, NGO's) are close enough to the organising principles of corporations that they could be regarded in the same, broad, social categorisation. His 18th century definition of 'Corporation' fits perfectly with the notion of 'Petit Bourgeois' of history - minor land-holding families, technical guilds, larger artisanal groups. Do we think Athenian 'Trireme' warships were built by a guy and his assistants? No, these were built by well organised groups with specialisation, paid for their work, aka 'Corporations'. The most interesting thing about these talks are actually social: what kinds of ideas and memes will rise to the fore among people who are aspirationally antagonising, like social anarchists (I don't mean that in a negative sense, I mean that would be the closest technical 'ideal' that defines the a group like 'Chaos Computer Club'). Mohawks, anti-estabishment statements such as 'ardent atheist', and hints of softly anti-corporate ideals. "Life-span of corporation is shorter, largely due to predation (totally unsubstantiated), corporation's are cannibals, they eat one another (that's an interesting way to describe mergers)." "For the first century and a 1/2 they depended entirely on human employees" - no, actually, the Industrial Revolution was literally about harnessing the power of fossil fuels via machines to automate that which would have been done by people (or animals) before. Neither humans nor horses moved those trains. And then of course the dehumanising of governments and corporations as AI: "What do our AI Overlords want?". I don't believe there is really any relationship between 'corporations' and 'AI', it's a neat idea. What we have here is an intelligent, creative guy with an antagonists worldview, working in a field wherein he's free to make up loose associations and hint at them as they are facts, ad he's subsumed an interesting ideal from CS into his own world view. I mean, it's great to try ideas that bends our minds a little bit, but I think it's clear he's a writer of fiction.
- einpoklum 6y ago
  Corporations are literally the something-extra over a "group of actors working together".
seamyb88 6y ago
> It's not like I'm predicting that airliners will fly slower and Nazis will take over the United States, is it? The latter wouldn't be a prediction in 2017, but a reflection of current affairs.
nojs 6y ago
The comparison between corporations optimising for profit to algorithms optimising for engagement, and the associated mispriced externalities in both cases being similar, is interesting. But otherwise this reads as a cookie-cutter rant about evil corporations and *ist AI without much deeper insight.
nemothekid 6y ago
Comparing Corporation to Sci-Fi super AI's is missing the point I think. I don't think the author literally believe corporations are AGIs, he starts with the point that corporations are "old, slow AIs". The article starts from a single argument - we have to be careful with AIs that are smarter than us, because if we don't instruct them carefully, we will end up in a paperclip maximizer situation. From there I take away 2 main points: 1. We do a shit job today of controlling "old, slow" paperclip maximizers, so there's no confidence we will ever be any better controlling an AI, despite any good intentions. 2. Our wild exotic ideas about paperclip maximizers probably won't come to fruition and instead we will end up in a boring dystopia where AIs will maximize time spent playing Farmville on Facebook.
- aaron695 6y ago
  > I don't think the author literally believe corporations are AGIs In the video questions section he confirms he means literal. https://www.youtube.com/watch?time_continue=3172&v=RmIgJ64z6Y4 https://www.youtube.com/watch?time_continue=3172&v=RmIgJ64z6... Look at HN as an AI, you can ask a very complex question and get a very complex answer, it's AGI. 4chan can send humans out to do its real world physical bidding. It can literally get people shot. These are very complex AI's.
  - hoseja 6y ago
    There is a difference between AI and a meme I think. AI is understood to run on computers, memes run on human brains. Corporations, nations, 4chan are memes, not AI. I just think this is a really neat idea.
    - aaron695 6y ago
      Ignoring Charlie Stross's definition, I would normally call HN emergent behaviour over AI. Perhaps off topic, but clearly memes have also changed our hardware. Old memes like religion will have definitely changed our evolution aka DNA and probably to make our DNA fit better with the meme. Corporations perhaps don't have long enough on a person lineage to make the change, but it'd be interesting to investigate.
Animats 6y ago
A point I've made on here before is the big near-term risk from AI is not general-purpose artificial intelligence. It's machine learning systems that do a better job of making corporate decisions than humans. Corporations are shareholder value maximizers. There's a powerful school of thought, the Chicago School, which claims that's all they should be, and have no other responsibilities. Machine learning systems are really good at maximizing some defined criterion. It's quite possible that they might get good at making corporate decisions. They already do that for some investment funds. Once machine learning systems are better at corporate decision making than humans, market forces will demand they be put in charge. The companies with inferior human-based technology will start to lose out. That's implicit in the forces behind capitalism. Be afraid, CEOs. Be very afraid. The machines are coming to take your job. (Somebody should turn this into a TED talk.)
- dleslie 6y ago
  Be afraid, humanity. All that stands between humanity and the short-term exploitation in unregulated niches where there are long-term negative externalities is human oversight with its pesky feelings. And even that fails us too often.
- koheripbal 6y ago
  > Corporations are shareholder value maximizers. The logic goes that organizations, in general, should focus on what they are good at, while the government should pass regulation to incentivize socially good behaviors. What we need to work on is having a stronger government that's less influenced (monetarily) by corporations, to set boundaries, set incentives, and police corporations to good behavior. To be honest, we often highlight all the places this goes wrong, but all the industries that don't get a lot of press are good examples of this working well.
hnick 6y ago
For anyone with some hours to burn, there is an idle game that lets you play the role of a paperclip maximiser. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html I find idle games odd. My rational mind knows they are traps, but I still want to see what is next. They tap into some kind of need for exploration, progression, and learning. This one at least explores a few ideas and has a set endpoint so I enjoyed going through it.
- nevster 6y ago
  And even though it has an end point, when you get to the end you want to do it again because you get bonuses...
- EricE 6y ago
  >I find idle games odd. Dopamine traps. To bring back to the article, there's a decent ST:NG episode where Wesley saves the day...
- uep 6y ago
  I greatly enjoyed this game because of its relationship to the thought experiment. Last time I tried playing it, it had a terrible memory leak on Firefox though. Eventually consuming over 15GB, when I decided I had to kill it.
darepublic 6y ago
When I predict what I'm going to accomplish in a given day I often get it wrong. Even in my own life I have a hard time predicting what will come next
netcan 6y ago
Reductio ad deus is a recurring pattern, or at least it was, when we look hopefully into a radical future. Radical political movements develop messianic themes, whether or not they rejected pre-existing ideas of god. Modern futurists are distinctly messianic. Remember also that (radical) 19th century politics kind of were futurist movements. "Singularity" is an on-the-nose example. ^Radical meaning "want or expect major societal change." I think we're better at predicting the future than we give ourselves credit for. We're just bad at distinguishing profound from banal in those futures. 100 years ago, economists and intellectuals (famously keynes) used their projections of productivity, technology & such to predict a leisure society. 15hr workweeks, etc. They were right about almost everything, except the conclusions. Productivity, global trade, technology, even peace... eventually. Even with the benefit of hindsight, very few modern economists reach profound conclusions about the mistakes of their forebears. The way we usually get the future wrong is "you were right, an yet..." The cultural element is the wild card. In 1990 you could have predicted 2020's radically changed landscape of media, social media, communication technologies & such. You probably couldn't have predicted the memetic influence on, the economy, education, social life, etc... At least, people usually don't predict these well.
polynomial 6y ago
> We're living in yesterday's future, and it's nothing like the speculations of our authors and film/TV producers. Exception proves the rule: http://www.openculture.com/2020/07/a-1947-french-film-accurately-predicted-our-21st-century-addiction-to-smartphones.html http://www.openculture.com/2020/07/a-1947-french-film-accura...
strken 6y ago
One of the biggest problems with thought experiments designed as an introduction to an entire class of problems, is that they get misinterpreted as a complete statement of the problem. Charlie Stross is quite dismissive of paperclip maximisers: Elon Musk "has an obsessive fear" of them, and "isn't paying enough attention" because Tesla does the same thing. He refers to a "pure paperclip maximiser" and discusses the "naive vision of a paperclip maximiser". This is quite insulting, because the paperclip maximiser is a thought experiment designed to introduce the consequences of intelligences which have fundamentally different values to human beings, in an accessible way. What he's doing is like reading a kids' book on counting and then writing an article contrasting naive apple-based addition with his brilliant new idea of generic fruit-based counting.

243 points by thanhhaimai 6y ago

(2017)

Not so sure about his future prediction ability. This, for example, like most "the sky is falling", didn't age well: https://www.antipope.org/charlie/blog-static/2018/07/that-sinking-feeling.html https://www.antipope.org/charlie/blog-static/2018/07/that-si...

nikanj 6y ago
Brexit hasn't happened yet. The transition period ends in about six months, so it's too early to say if he was right or wrong
klenwell 6y ago
It looks to me like most Stross's predictions depend first on the event of the Brexit, which has not occurred yet as far as I know. Here's a map the NY Times published on 6/15/2005: http://graphics8.nytimes.com/images/2005/06/15/business/arm3.gif http://graphics8.nytimes.com/images/2005/06/15/business/arm3... It makes an implicit prediction about the then booming housing market and the sky above it. On that date, the S&P 500 closed at 1,206.58. Two years later, in 2007, the S&P 500 closed at 1,522.97. To the extent that stock markets tell us where we stand as a nation or a society, I find this example instructive. There's such a thing as the long view. It's hard to know how long it should be. Harder to know whether it will be proved out in the end until it is proved or disproved. Even then sometimes we miss the forest for the trees. S&P Data: https://finance.yahoo.com/quote/%5EGSPC/history?period1=1118793600&period2=1181865600&interval=1d&filter=history&frequency=1d https://finance.yahoo.com/quote/%5EGSPC/history?period1=1118...
rsynnott 6y ago
Remember that Brexit hasn’t, for practical purposes, happened yet. Most people didn’t anticipate that the UK would request, and be granted, so many extensions, a few years ago. However, time has now run out; the last realistic opportunity for a further extension passed a couple of weeks ago, and Brexit will now happen in some real sense at the end of the year. It likely won’t be pretty.

deleted 6y ago

[deleted]

dang 6y ago

If curious see also 2018 https://news.ycombinator.com/item?id=16051337 https://news.ycombinator.com/item?id=16051337 2017 https://news.ycombinator.com/item?id=16032643 https://news.ycombinator.com/item?id=16032643

jengleton 6y ago

Robert Miles has an interesting response to the claim that corporations are AIs: https://www.youtube.com/watch?v=L5pUA3LsEaw https://www.youtube.com/watch?v=L5pUA3LsEaw To paraphrase: yes, corporations function as agents, but their maximum performance is limited by the capabilities of their employees. A corporate AI may have a much broader set of skills than any single person, and may be able to tackle many concurrent tasks, but as an "intelligent agent" its decision making capabilities probably don't scale exponentially (or even linearly) with its headcount. In the sense that a corporation's maximum intelligence is likely to be in the same ballpark as the smartest humans, it can't be seen as a true superintelligence.

zaphar 6y ago
I don't think the comparison was to AI as a super intelligence. AI doesn't have to be a super intelligence to be AI or to be dangerous.
- WrtCdEvrydy 6y ago
  I think AS (Artificial Stupidity) will get us first.
  - nekopa 6y ago
    Just NO. Artificial Stupidity (AS) will never be able to perform at the level of Actually Seen Stupidity (ASS). No matter how much you try to improve AS, God will always create a better ASS. You will always be limited by priors to create an AS, but ASS will evolve naturally, plus if God ever decides to use science and apply a Directed Acyclic Tangent (DAT) well, DAT ASS would be unbeatable.
- jengleton 6y ago
  That's fair. I interpreted the comparison as being between "AI implemented as a property of human organizations" and "AI implemented as a powerful search algorithm". While corporations can certainly be dangerous, they're still made of people and thus are unlikely to want to do things like "convert the entire mass of the solar system into dollar bills" (and even if they did, they'd have a hard time doing it). A sufficiently powerful search algorithm would find all sorts of bizarre ways of satisfying its goals. The point is that the scope of the risk is quite different when talking about corporations (order of magnitude: screwing up the environment in pursuit of easy profit) vs "real" AI which, if granted agency, would have potentially unbounded risk. Another way to look at it is with regards to capacity for self-modification. If a corporation can't re-structure itself into being much smarter than the smartest human, its intelligence is fundamentally limited, and therefore so is the risk. Does software have this restriction? We don't really know yet, but it's hard to point to exactly why it would.

anchpop 6y ago

> And once you start probing the nether regions of transhumanist thought and run into concepts like Roko's Basilisk—by the way, any of you who didn't know about the Basilisk before are now doomed to an eternity in AI hell—you realize they've mangled it to match some of the nastiest ideas in Presybterian Protestantism. I wish every discussion of transhumanism didn't have to involve Roko's Basilisk. It's not something anyone takes seriously (and very few ever did), but it has enough quirky weirdness that everyone seems to want to talk about it. Here's a quote from "Lesswrongwiki": > Roko's argument was broadly rejected on Less Wrong, with commenters objecting that an agent like the one Roko was describing would have no real reason to follow through on its threat > [...] > Less Wrong's founder, Eliezer Yudkowsky, banned discussion of Roko's basilisk on the blog for several years as part of a general site policy against spreading potential information hazards. This had the opposite of its intended effect: a number of outside websites began sharing information about Roko's basilisk, as the ban attracted attention to this taboo topic. Websites like RationalWiki spread the assumption that Roko's basilisk had been banned because Less Wrong users accepted the argument; thus many criticisms of Less Wrong cite Roko's basilisk as evidence that the site's users have unconventional and wrong-headed beliefs. - https://wiki.lesswrong.com/wiki/Roko's_basilisk https://wiki.lesswrong.com/wiki/Roko's_basilisk

scotty79 6y ago
For me this illustrates nicely why most discussions about AI and singularity are like counting how many angels can dance on the head of a pin. It's all like talking about nuclear weapons in 17th century. Sure you can but even if you reach some conclusions they most likely will be invalidated by first technical detail of the implementation of thing when it actually gets created.
LiquidSky 6y ago
Your comment (and your links) miss the point of mocking references to "Roko's Basilisk". Whether it's actually a widely-held belief of the Less Wrong/greater rationalist community is irrelevant, it's emblematic of the eye-rolling nonsense that they engage in. It's the exact secular equivalent of "How many angels can dance on the head of a pin?"[1], which was also not an actual accepted topic of medieval religious scholarly debate but an illustration of the usual absurdity of such debate (e.g., there was an actual centuries-long scholarly discussion of whether angels were sexless or had sexes). [1]https://en.wikipedia.org/wiki/How_many_angels_can_dance_on_the_head_of_a_pin%3F https://en.wikipedia.org/wiki/How_many_angels_can_dance_on_t...

autokad 6y ago

>"Science fiction is written by people embedded within a society with expectations and political assumptions that bias us towards looking at the shiny surface of new technologies rather than asking how human beings will use them, and to taking narratives of progress at face value rather than asking what hidden agenda they serve." And be prepared for the most agenda ridden text you have read in a long time.

claudiawerner 6y ago
That's not necessarily a bad thing; in fact, most works (and specifically blogs) online have an "agenda".
- Nasrudith 6y ago
  Not having any agenda is a deeply unhealthy mental state as it implies a complete apathy to all outcomes. It is just that most "normal" agendas are effectively invisible as a Huffman encoded to a 0 bit default. Not wanting to be tortured to death is an agenda technically but contributes so little information that it only makes sense to state the opposite.
thiagocsf 6y ago
Would you mind explaining what the agenda is?
danielscrubs 6y ago
He was also all over the place. But interesting ideas and if he can connect the ideas in a better way in his book I wouldn’t mind reading it.
loa_in_ 6y ago
That's absolutely not what I associate science fiction with. Stanisław Lem for example couldn't be further from taking shiny technology at face value
- aasasd 6y ago

bawolff 6y ago

> Nobody in 2007 was expecting a Nazi revival in 2017, right? Is that actually true? I seem to remember people predicting that usa has been edging towards facism for a while now, especially in the aftermath of 9/11. For example here is an article from 2003 predicting usa is heading towards fasicism https://secularhumanism.org/2003/03/fascism-anyone/ https://secularhumanism.org/2003/03/fascism-anyone/

bitwize 6y ago
Nobody among the intellectual class anticipated Trump's election in 2016. Much like physicists hypothesized dark matter to account for the apparent greater mass necessary to stabilize galaxies and such, the political intelligentsia hypothesized an enormous, hidden body of "dark Nazis" to account for their chosen candidate not winning the election. It is unfathomable to them that enough people in the center, center-right, and right could be discontent enough with being told whom to vote for and why someone running on a platform of tighter immigration controls and more jobs for Americans is evil incarnate, that they were willing to "hold their nose and vote for Trump", to swing the election in his favor. Political intellectuals tend to lean left, and whenever a rightist scores a major political victory they start predicting stormtroopers goosestepping through American streets Real Soon Now, going back to at least Reagan. So the Nazi Revival is generally accepted as real, and if you contest the idea's truth you may be considered one of them.
- jrumbut 6y ago
  This is very untrue, even if we take nobody as a little overstatement, half of the election prediction models had him above 10% chance of winning (FiveThirtyEight had him at 29%, https://www.nytimes.com/interactive/2016/upshot/presidential-polls-forecast.html?_r=0#other-forecasts https://www.nytimes.com/interactive/2016/upshot/presidential...). WBUR, Boston's NPR news station, published in their blog "Cognoscenti" an article titled "Why Donald Trump will Win in November" in May of 2016 (https://www.wbur.org/cognoscenti/2016/05/06/election2016-trump-establishment-politics-tom-keane https://www.wbur.org/cognoscenti/2016/05/06/election2016-tru...). I can't think of anything more representative of what people think of as the left leaning intellectual class than a Boston NPR blog with a Latin title. This idea that him winning in 2016 was inconceivable to media/intellectual circles is a bit of revisionist history.

Ygg2 6y ago

> nastiest ideas in Presybterian Protestantism. And if you follow Charlie's reasoning, that simpler parts, can't lead to a highly advanced intelligence, you come to conclusion intelligence is derived from a divine soul. Intelligence isn't as uncommon as we think, but it takes long time to emerge.

hwillis 6y ago
> if you follow Charlie's reasoning, that simpler parts, can't lead to a highly advanced intelligence, you come to conclusion intelligence is derived from a divine soul. Thats an absolutely ridiculous interpretation. Asserting that our current "simple parts" are still very far from AGI is nowhere even close to asserting that intelligence is immaterial. Its very, very obvious that there are myriad possible leaps we might need to get closer to intelligence. You can't build an attention network from a perceptron. Its totally reasonable to say that we need a few more fundamental discoveries first. Obvious, even, given the insane amount of processing required to teach a network a language. Brains are clearly wired in a way that is simply more efficient than how we currently know how to build stuff.
- Ygg2 6y ago
  > Thats an absolutely ridiculous interpretation. Asserting that our current "simple parts" are still very far from AGI is nowhere even close to asserting that intelligence is immaterial. Not really what it says. Look at following passage: > AI singularity as a narrative, and identify the numerous places in the story where the phrase "... and then a miracle happens" occurs, it becomes apparent pretty quickly that they've reinvented Christianity. The phrase "miracle happens" , to me is suspect. Intelligence rose many times in various creatures. There is nothing miracle about something that rose multiple number of times. Are Singularists wrong? Yes. They confuse saturation for exponential curves; current neural networks are far cry from actual neuron networks, and their time scales reflect more their fear of death than any sensible timeline. If Charlie wants to criticize Singularists, there are plenty of valid reasons. Them being cult like is the least important one.
  - 6y ago

ALittleLight 6y ago

I found a couple errors early on that dissuaded me from continuing to read. One was the idea that the author could rule out the singularity because he wasn't aware of progress toward self-motivated AI that would be something like our own intelligence. This seems like a limited view to me because we wouldn't need self-motivation at all in AI to hit the singularity. Humans can supply the self motivation. Suppose I'm using GPT-4 or GPT-44 trained on the corpus from sci-hub and it recommends experiments to me, or explains physics to me, etc. I could be the self-motivating part and the AI could be the intelligence part, and it seems we'd still hit the technological singularity. Another problem I had was when the author characterized Elon Musk's "obsession" with the paperclip maximizer and described Tesla as a battery maximizer. It seems like the author kind of misses the point of the paperclip thought experiment, which is, broadly, that an AI's interests might not be aligned with our own and that misalignment may cause serious problems. Tesla is clearly not a battery maximizer and it is clearly not a different class of intelligence from the humans and corporations existing today (though it may be towards the top of that class). Neither of those things would necessarily be true of an AI. Given the position of my scrollbar it seems I was only starting to read this piece, but already finding what I think are significant problems as the author sets up the argument, I'm hesitant to spend more reading.

worldsayshi 6y ago
> Tesla [...] is clearly not a different class of intelligence from [...] corporations I believe he just used Tesla as an example of corporation not as something new or special. > the point of the paperclip thought experiment is, broadly, that an AI's interests might not be aligned with our own and that misalignment may cause serious problems. But (some) corporations clearly do share this aspect to some extent. They maximize profit while environmental concerns are not prioritized. And he pointed out later that corporations are slightly more complex than just maximizing 'one' thing. But a range of things and most importantly profit.
- ALittleLight 6y ago
  Most, if not all, corporations are misaligned to humanity's well-being. That is, they want money, and that's not exactly the same thing as what would be best for humanity. This is okay though because corporations are the same class and kind of intelligence as the rest of humanity and so they can be (somewhat) predicted and constrained by our laws and norms. Because corporations are the same class of intelligence, they can't outsmart everyone. We can catch them when they do wrong and punish them and use laws and courts to control them. Because corporations are the same kind of intelligence they (usually) aren't going to do things we might consider insane or sociopathic. For example, if corporation X realized they might make a profit by taking out life insurance on workers and then getting them to do very risky things, X Corp probably wouldn't do it and if they did people would leak it and laws would be made to stop it. The paperclip thought experiment is an illustration of a different kind of intelligence that wouldn't be constrained by human morality or norms and would surpass human intelligence and not be constrained by laws or force either.

jariel 6y ago

Corporations are nothing but a group of actors working together to achieve some outcome, part of which may be profit, and it's absurd to think that they are a 'Modern Era', or even Western concept as every single land-owning farmer, artisan, banker, merchant in history was de-facto a form of 'corporation'. Moreover, many institutions with origins in antiquity (schools, governments, NGO's) are close enough to the organising principles of corporations that they could be regarded in the same, broad, social categorisation. His 18th century definition of 'Corporation' fits perfectly with the notion of 'Petit Bourgeois' of history - minor land-holding families, technical guilds, larger artisanal groups. Do we think Athenian 'Trireme' warships were built by a guy and his assistants? No, these were built by well organised groups with specialisation, paid for their work, aka 'Corporations'. The most interesting thing about these talks are actually social: what kinds of ideas and memes will rise to the fore among people who are aspirationally antagonising, like social anarchists (I don't mean that in a negative sense, I mean that would be the closest technical 'ideal' that defines the a group like 'Chaos Computer Club'). Mohawks, anti-estabishment statements such as 'ardent atheist', and hints of softly anti-corporate ideals. "Life-span of corporation is shorter, largely due to predation (totally unsubstantiated), corporation's are cannibals, they eat one another (that's an interesting way to describe mergers)." "For the first century and a 1/2 they depended entirely on human employees" - no, actually, the Industrial Revolution was literally about harnessing the power of fossil fuels via machines to automate that which would have been done by people (or animals) before. Neither humans nor horses moved those trains. And then of course the dehumanising of governments and corporations as AI: "What do our AI Overlords want?". I don't believe there is really any relationship between 'corporations' and 'AI', it's a neat idea. What we have here is an intelligent, creative guy with an antagonists worldview, working in a field wherein he's free to make up loose associations and hint at them as they are facts, ad he's subsumed an interesting ideal from CS into his own world view. I mean, it's great to try ideas that bends our minds a little bit, but I think it's clear he's a writer of fiction.

einpoklum 6y ago
Corporations are literally the something-extra over a "group of actors working together".

seamyb88 6y ago

> It's not like I'm predicting that airliners will fly slower and Nazis will take over the United States, is it? The latter wouldn't be a prediction in 2017, but a reflection of current affairs.

nojs 6y ago

The comparison between corporations optimising for profit to algorithms optimising for engagement, and the associated mispriced externalities in both cases being similar, is interesting. But otherwise this reads as a cookie-cutter rant about evil corporations and *ist AI without much deeper insight.

nemothekid 6y ago

Comparing Corporation to Sci-Fi super AI's is missing the point I think. I don't think the author literally believe corporations are AGIs, he starts with the point that corporations are "old, slow AIs". The article starts from a single argument - we have to be careful with AIs that are smarter than us, because if we don't instruct them carefully, we will end up in a paperclip maximizer situation. From there I take away 2 main points: 1. We do a shit job today of controlling "old, slow" paperclip maximizers, so there's no confidence we will ever be any better controlling an AI, despite any good intentions. 2. Our wild exotic ideas about paperclip maximizers probably won't come to fruition and instead we will end up in a boring dystopia where AIs will maximize time spent playing Farmville on Facebook.

aaron695 6y ago
> I don't think the author literally believe corporations are AGIs In the video questions section he confirms he means literal. https://www.youtube.com/watch?time_continue=3172&v=RmIgJ64z6Y4 https://www.youtube.com/watch?time_continue=3172&v=RmIgJ64z6... Look at HN as an AI, you can ask a very complex question and get a very complex answer, it's AGI. 4chan can send humans out to do its real world physical bidding. It can literally get people shot. These are very complex AI's.
- hoseja 6y ago
  There is a difference between AI and a meme I think. AI is understood to run on computers, memes run on human brains. Corporations, nations, 4chan are memes, not AI. I just think this is a really neat idea.
  - aaron695 6y ago
    Ignoring Charlie Stross's definition, I would normally call HN emergent behaviour over AI. Perhaps off topic, but clearly memes have also changed our hardware. Old memes like religion will have definitely changed our evolution aka DNA and probably to make our DNA fit better with the meme. Corporations perhaps don't have long enough on a person lineage to make the change, but it'd be interesting to investigate.

Animats 6y ago

A point I've made on here before is the big near-term risk from AI is not general-purpose artificial intelligence. It's machine learning systems that do a better job of making corporate decisions than humans. Corporations are shareholder value maximizers. There's a powerful school of thought, the Chicago School, which claims that's all they should be, and have no other responsibilities. Machine learning systems are really good at maximizing some defined criterion. It's quite possible that they might get good at making corporate decisions. They already do that for some investment funds. Once machine learning systems are better at corporate decision making than humans, market forces will demand they be put in charge. The companies with inferior human-based technology will start to lose out. That's implicit in the forces behind capitalism. Be afraid, CEOs. Be very afraid. The machines are coming to take your job. (Somebody should turn this into a TED talk.)

dleslie 6y ago
Be afraid, humanity. All that stands between humanity and the short-term exploitation in unregulated niches where there are long-term negative externalities is human oversight with its pesky feelings. And even that fails us too often.
koheripbal 6y ago
> Corporations are shareholder value maximizers. The logic goes that organizations, in general, should focus on what they are good at, while the government should pass regulation to incentivize socially good behaviors. What we need to work on is having a stronger government that's less influenced (monetarily) by corporations, to set boundaries, set incentives, and police corporations to good behavior. To be honest, we often highlight all the places this goes wrong, but all the industries that don't get a lot of press are good examples of this working well.

hnick 6y ago

For anyone with some hours to burn, there is an idle game that lets you play the role of a paperclip maximiser. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html I find idle games odd. My rational mind knows they are traps, but I still want to see what is next. They tap into some kind of need for exploration, progression, and learning. This one at least explores a few ideas and has a set endpoint so I enjoyed going through it.

nevster 6y ago
And even though it has an end point, when you get to the end you want to do it again because you get bonuses...
EricE 6y ago
>I find idle games odd. Dopamine traps. To bring back to the article, there's a decent ST:NG episode where Wesley saves the day...
uep 6y ago
I greatly enjoyed this game because of its relationship to the thought experiment. Last time I tried playing it, it had a terrible memory leak on Firefox though. Eventually consuming over 15GB, when I decided I had to kill it.

darepublic 6y ago

When I predict what I'm going to accomplish in a given day I often get it wrong. Even in my own life I have a hard time predicting what will come next

netcan 6y ago

Reductio ad deus is a recurring pattern, or at least it was, when we look hopefully into a radical future. Radical political movements develop messianic themes, whether or not they rejected pre-existing ideas of god. Modern futurists are distinctly messianic. Remember also that (radical) 19th century politics kind of were futurist movements. "Singularity" is an on-the-nose example. ^Radical meaning "want or expect major societal change." I think we're better at predicting the future than we give ourselves credit for. We're just bad at distinguishing profound from banal in those futures. 100 years ago, economists and intellectuals (famously keynes) used their projections of productivity, technology & such to predict a leisure society. 15hr workweeks, etc. They were right about almost everything, except the conclusions. Productivity, global trade, technology, even peace... eventually. Even with the benefit of hindsight, very few modern economists reach profound conclusions about the mistakes of their forebears. The way we usually get the future wrong is "you were right, an yet..." The cultural element is the wild card. In 1990 you could have predicted 2020's radically changed landscape of media, social media, communication technologies & such. You probably couldn't have predicted the memetic influence on, the economy, education, social life, etc... At least, people usually don't predict these well.

polynomial 6y ago

> We're living in yesterday's future, and it's nothing like the speculations of our authors and film/TV producers. Exception proves the rule: http://www.openculture.com/2020/07/a-1947-french-film-accurately-predicted-our-21st-century-addiction-to-smartphones.html http://www.openculture.com/2020/07/a-1947-french-film-accura...

strken 6y ago

One of the biggest problems with thought experiments designed as an introduction to an entire class of problems, is that they get misinterpreted as a complete statement of the problem. Charlie Stross is quite dismissive of paperclip maximisers: Elon Musk "has an obsessive fear" of them, and "isn't paying enough attention" because Tesla does the same thing. He refers to a "pure paperclip maximiser" and discusses the "naive vision of a paperclip maximiser". This is quite insulting, because the paperclip maximiser is a thought experiment designed to introduce the consequences of intelligences which have fundamentally different values to human beings, in an accessible way. What he's doing is like reading a kids' book on counting and then writing an article contrasting naive apple-based addition with his brilliant new idea of generic fruit-based counting.