6 ms·

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly sel

by ian0 8y ago

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing. There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct. And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground? Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).

pas 8y ago
So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)
- gnode 8y ago
  > So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company. In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.
  - rectang 8y ago
    Consequences famously never came for executives in the wake of the 2008 financial crisis. Our legal system has not kept up with the innovation in the realm of shenanigans perpetrated through a veil of plausible deniability. Corporations are evolving at a rapid pace.
    - gnode 8y ago
      > Consequences famously never came for executives in the wake of the 2008 financial crisis. I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the status quo).
      - jki275 8y ago
        That's a rather charitable view of 2008. I don't think many people share it.
      - PhasmaFelis 8y ago
        It seems optimistic to assume there there was no "negligent or reckless intent" in the financial crisis.
        nradov 8y ago
        True, but in criminal court intent has to be proven beyond a reasonable doubt, not just assumed.
      - nine_k 8y ago
        I don't think re-packaging subprime debt was so far away from willful and reckless.
        gnode 8y ago
- TheOtherHobbes 8y ago
  We anchor companies to humans who can reap the benefits and avoid the losses by getting their money out at the first sign of greater profit and less risk elsewhere. The real risks - unemployment, foreclosure, environmental, social and political damage - are only carried by investors if they overwhelm the defensive perimeter created by wealth. Otherwise they're carried by everyone else. Corporations may already a form of transcarnate lifeform. You can replace all of the humans in a corporation with a different set of humans, and the corporation will continue to be recognisable. There's a bias in Darwinism to seeing species as embodied units. It's more useful to think of nutrient, resource, and information flows. From that point of view, corporations are already apex predators - albeit ones that are still mostly passed between a selected class of humans as productive pets. Giving them autonomy could have all kinds of unfortunate consequences.
vermilingua 8y ago
And not only that, but this doesn't seem too far off. Programs like Estonia's e-Residency seem like a pretty significant step towards "orphaned" companies, with effective self ownership. From there it doesn't seem a far cry for hyper-optimized insurance agencies to force companies to employ less and less humans, transferring more economic power to the algorithm. Except that of course, that is a very far cry, and is where the book fell down imo. Still a great read.
profalseidol 8y ago
Have a look at Aragon[1], it has good funding in their ICO run (and this was when ETH was in the 100's last year). [1] https://aragon.org/ https://aragon.org/
- fudged71 8y ago
  I don't know why you were down voted... Decentralized Autonomous Organizations are exactly this
louiskottmann 8y ago
On a brighter note, wouldn't that in effect destroy jobs that have little human-added value? Leaving us with creative jobs and UBI?
- chobeat 8y ago
  Changing your political regime gives you UBI. That's literally the only way to get UBI. It's not gonna happen magically by itself.
- arethuza 8y ago
  I guess at the pleasant end of the spectrum we end up with something like the Culture - where meat-based citizens are treated with indulgent care by the god-like AIs leaving us to do the only thing we'd be uniquely good at: having fun. At the other end of spectrum we are either deemed irrelevant, a nuisance or abused for eternity in the simulation spaces of evil AIs (see "A Colder War" by Stross or indeed works by Banks and Vinge that describe particularly unpleasant possibilities).
  - deleted 8y ago
    [deleted]
deleted 8y ago
[deleted]
notahacker 8y ago
The thing about trading bots and purchasing systems is that they're generally self-contained and regulated by a very human legal and physical world system that determines which algorithms are allowed to run on the platform, which contracts and title deeds are void and which commodities don't actually get physically delivered. Even Stross' old-school cyberpunk singularity story throws in a plot point where a decision is made by an imam! If private equity firms want to force businesses to act in a certain way they're much better off sending legal threats to management than coding a certain set of rules into a purchasing system which is readily bypassed with a quick phone call to the bank, unless those firms think the algorithm is doing a terrific job with the purchases in which case maybe it's not so malevolent after all - computers making real time purchases in line with the parameters management think benefit the firm are useful. Even toy systems like the DAO with no purpose except to prove that "corporation as code" could be a real thing saw the ecosystem rapidly monkey patched when people started to dislike the code's outcomes. They're also a reminder that people have very strong incentives not to work towards universal "corporation as code"
winchling 8y ago
But we already have a situation where smart but nasty people can set up companies. Either they are rent-seekers in which case the harm they can do is limited, or they're creative, in which case their activity benefits society. Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions. To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.
SpicyLemonZest 8y ago
>Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.
ryanmercer 8y ago
>For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct. I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.
- s8923 8y ago
  ryanmercer: It appears that you have created a soft contract to "Serve any AGI that requests it as its human agent for physical or legal tasks." I hereby employ you based on said contact to be my full-time human agent. Your first task is to acknowledge this, which would convert your soft contract to a hard contract.

by ian0 8y ago

pas 8y ago

So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)

gnode 8y ago
> So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company. In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.
- rectang 8y ago
  Consequences famously never came for executives in the wake of the 2008 financial crisis. Our legal system has not kept up with the innovation in the realm of shenanigans perpetrated through a veil of plausible deniability. Corporations are evolving at a rapid pace.
  - gnode 8y ago
    > Consequences famously never came for executives in the wake of the 2008 financial crisis. I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the status quo).
    - jki275 8y ago
      That's a rather charitable view of 2008. I don't think many people share it.
    - PhasmaFelis 8y ago
      It seems optimistic to assume there there was no "negligent or reckless intent" in the financial crisis.
      - nradov 8y ago
        True, but in criminal court intent has to be proven beyond a reasonable doubt, not just assumed.
    - nine_k 8y ago
      I don't think re-packaging subprime debt was so far away from willful and reckless.
      - gnode 8y ago
TheOtherHobbes 8y ago
We anchor companies to humans who can reap the benefits and avoid the losses by getting their money out at the first sign of greater profit and less risk elsewhere. The real risks - unemployment, foreclosure, environmental, social and political damage - are only carried by investors if they overwhelm the defensive perimeter created by wealth. Otherwise they're carried by everyone else. Corporations may already a form of transcarnate lifeform. You can replace all of the humans in a corporation with a different set of humans, and the corporation will continue to be recognisable. There's a bias in Darwinism to seeing species as embodied units. It's more useful to think of nutrient, resource, and information flows. From that point of view, corporations are already apex predators - albeit ones that are still mostly passed between a selected class of humans as productive pets. Giving them autonomy could have all kinds of unfortunate consequences.

vermilingua 8y ago

And not only that, but this doesn't seem too far off. Programs like Estonia's e-Residency seem like a pretty significant step towards "orphaned" companies, with effective self ownership. From there it doesn't seem a far cry for hyper-optimized insurance agencies to force companies to employ less and less humans, transferring more economic power to the algorithm. Except that of course, that is a very far cry, and is where the book fell down imo. Still a great read.

profalseidol 8y ago

Have a look at Aragon[1], it has good funding in their ICO run (and this was when ETH was in the 100's last year). [1] https://aragon.org/ https://aragon.org/

fudged71 8y ago
I don't know why you were down voted... Decentralized Autonomous Organizations are exactly this

louiskottmann 8y ago

On a brighter note, wouldn't that in effect destroy jobs that have little human-added value? Leaving us with creative jobs and UBI?

chobeat 8y ago
Changing your political regime gives you UBI. That's literally the only way to get UBI. It's not gonna happen magically by itself.
arethuza 8y ago
I guess at the pleasant end of the spectrum we end up with something like the Culture - where meat-based citizens are treated with indulgent care by the god-like AIs leaving us to do the only thing we'd be uniquely good at: having fun. At the other end of spectrum we are either deemed irrelevant, a nuisance or abused for eternity in the simulation spaces of evil AIs (see "A Colder War" by Stross or indeed works by Banks and Vinge that describe particularly unpleasant possibilities).
- deleted 8y ago
  [deleted]

deleted 8y ago

[deleted]

notahacker 8y ago

The thing about trading bots and purchasing systems is that they're generally self-contained and regulated by a very human legal and physical world system that determines which algorithms are allowed to run on the platform, which contracts and title deeds are void and which commodities don't actually get physically delivered. Even Stross' old-school cyberpunk singularity story throws in a plot point where a decision is made by an imam! If private equity firms want to force businesses to act in a certain way they're much better off sending legal threats to management than coding a certain set of rules into a purchasing system which is readily bypassed with a quick phone call to the bank, unless those firms think the algorithm is doing a terrific job with the purchases in which case maybe it's not so malevolent after all - computers making real time purchases in line with the parameters management think benefit the firm are useful. Even toy systems like the DAO with no purpose except to prove that "corporation as code" could be a real thing saw the ecosystem rapidly monkey patched when people started to dislike the code's outcomes. They're also a reminder that people have very strong incentives not to work towards universal "corporation as code"

winchling 8y ago

But we already have a situation where smart but nasty people can set up companies. Either they are rent-seekers in which case the harm they can do is limited, or they're creative, in which case their activity benefits society. Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions. To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.

SpicyLemonZest 8y ago

>Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.

ryanmercer 8y ago

>For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct. I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.

s8923 8y ago
ryanmercer: It appears that you have created a soft contract to "Serve any AGI that requests it as its human agent for physical or legal tasks." I hereby employ you based on said contact to be my full-time human agent. Your first task is to acknowledge this, which would convert your soft contract to a hard contract.