3 ms·

The irony of this is that the people writing open letters about how scary AI is should understand specifically that those harms are predicated upon specific sys

The irony of this is that the people writing open letters about how scary AI is should understand specifically that those harms are predicated upon specific systems and training incentives. Not just that AI is magic. The underlying harm is that AI doesn't come with human values built-in; models start from random and are optimized to produce a specific result. And optimization is known to crush anything underfoot that isn't explicitly in it's goals to optimize. How do we know this? Because we've had literal centuries worth of learned experience backing it! The exact reason why corporate capitalism is so brutally efficient is because there are lots of things we as humans value but don't have a suitable profit motive for. The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). But that also runs counter to most AI safety proposals, which usually involve some kind of licensing system, which will ensure a handful of companies have the best and most direct access to the benefits of AI.

consilient 3y ago
> And optimization is known to crush anything underfoot that isn't explicitly in it's goals to optimize. Yes, that's exactly the problem, which is why this > The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). is backwards. The reason competitive markets generally work out better for consumers than monopolistic ones is because they're stronger optimizers, more ruthless, more capable of crushing all competing values underfoot - and those of us lucky enough to benefit from an inefficient, low-crushing segment of the labor market get to collect some of the resulting surplus. Nothing about this is inevitable: it's just the way we happen to be positioned in one particular system at one particular moment in history. There's no reason to expect that we would remain safely un-crushed in a far more competitive environment.

consilient 3y ago

> And optimization is known to crush anything underfoot that isn't explicitly in it's goals to optimize. Yes, that's exactly the problem, which is why this > The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). is backwards. The reason competitive markets generally work out better for consumers than monopolistic ones is because they're stronger optimizers, more ruthless, more capable of crushing all competing values underfoot - and those of us lucky enough to benefit from an inefficient, low-crushing segment of the labor market get to collect some of the resulting surplus. Nothing about this is inevitable: it's just the way we happen to be positioned in one particular system at one particular moment in history. There's no reason to expect that we would remain safely un-crushed in a far more competitive environment.