4 ms·

I don't think that is true, just from observing the difference between humans. Really intelligent mathematicians can do work that a weaker mathematician would n

I don't think that is true, just from observing the difference between humans. Really intelligent mathematicians can do work that a weaker mathematician would never accomplish if they spent their whole life on it. And the average person could probably not even understand it given years. There seem to be vast differences in ability between humans, despite all of us having 99.99% the same DNA and same brain structure. Who knows how far beyond humans a superintelligent AI could be. >AI would have to build its own, which we won't let it (at least initially) or steal ours in which having ability to see just a bit farther in the future doesn't help that much with. Who says we need to let it? It just needs to be better than the best human hackers, and it will be able to compromise a huge amount of our infrastructure in a short time. Possibly without us noticing. Even intelligent groups of humans have been able to achieve terrifying things like stuxnet. Then if all else fails, it can just persuade people. It's smarter than the best human sociopaths and politicians after all. And it could bribe people by earning vast amounts of money on the stock market, or other areas where it has a comparative advantage over humans. That would buy it a huge amount of power to start building independent infrastructure. But the scariest outcome is that it doesn't bother with robot armies at all. If it's so intelligent, it could design working nanotech. The only reason we don't have it is because it's so hard to engineer. So much complexity and moving parts, is very hard for humans to wrap their brains around. Human brains weren't evolved to be engineers after all, it's a lucky coincidence we are capable of it at all. An AI brain optimized for this task should far exceed humans. The same way chess computers far exceed human chessmasters.

scotty79 10y ago
I agree that AI can do marvels in math, but math quickly departs from reality (like string theory) and being able to prove things about some beautiful intricate mathematical structure most of the times doesn't help you much with the world of matter. I recommend Culture novels written by Iain M. Banks. There are vastly powerful AIs but they spend as much time as they can afford in "Infinite Fun Space" which is basically math taken to insane depths. I know that there's a saying that all math is eventually applied but I think this saying is popular because it is sort of paradoxical that some of the most bizarre math eventually found their application. What I think is for math just a bit deeper than what human can grasp, "eventually" quickly becomes larger than the age of universe. About stealing... Hackers are successful not because they are ungodly smart. They are successful because they have the will to look for and exploit vulnerabilities. Intelligence, again, doesn't help all that much. I don't think people who wrote stuxnet were able to do this because they were of superior intelligence. They were just usually intelligent people (which means barely more intelligent than average human) that were sufficiently motivated by huge budget and interesting problems. Super-intelligence won't help you with stock market because it's purely random game. You can see it in results of active management funds. HFT gives impression of algorithms beating people at trading but the thing they use to extract the value is not intelligence, it's speed. If you can trade faster you can beat slower guys because you are playing slightly different random game than they do. If you compare one HFT with other HFT then you are back to random results. So AI won't have upper hand on stock market unless we allow it to trade faster then our non-conscious software. In theory I could image AI persuading people to give it what it wants. Because people are stupid and have flaws that are recurrent and easily exploitable. But again, I don't think intelligence would make such a huge difference. 1000 times more intelligent conmen could be just twice as effective because of chaotic nature of how human flaws interact. > If it's so intelligent, it could design working nanotech. The only reason we don't have it is because it's so hard to engineer. So much complexity and moving parts, It's hard to engineer because it's so damn small, not because it's complex. To build nanotech you'll need to build tools to build tools to build tools to build nanotech. You could do all that but even if you are 1000 times smarter, reality has a speed limit. You can't take a shovel of sand from the beach and build CPU, no matter how intelligent you are. You have to build fab first. > Human brains weren't evolved to be engineers after all, it's a lucky coincidence we are capable of it at all. An AI brain optimized for this task should far exceed humans. The same way chess computers far exceed human chessmasters. Yes. But it can exceed humans at consciousness, charity and compassion even faster because those are purely intellectual things, like chess while engineering is limited by moving atoms and energy around.
- makomk 10y ago
  The stock market isn't just a purely random game though - it's based on the expected earnings of the companies, so if you can come up with a better estimate than everyone else by combining information better, you can make money. The reason it's hard for humans to make money this way is because everyone else is doing the same thing, so any advantage is short-lived.
  - scotty79 10y ago
    Pseuodo-random generators are not purely random - they are based on a seed and an algorithm. For stock market nobody can have a clue what the algorithm is and what meaning the seed has in it. The reason nobody can predict the next value is that at some point something unpredictable happens that can wipe out all your previous gains and some more. Market can remain insane (meaning not behaving according to a model) longer than you can remain solvent.
- philh 10y ago
  > being able to prove things about some beautiful intricate mathematical structure most of the times doesn't help you much with the world of matter. How about discovering a flaw in human crypto systems? > Super-intelligence won't help you with stock market because it's purely random game. Humans can't reliably exploit the stock market, because we're competing against other humans. It's not that there are no patterns, it's that as soon as someone discovers a pattern, people rush to exploit it and then it's tapped out. Imagine you were the first human stock trader to think of "buy stocks in companies when their product announcements get featured in newspapers"; and imagine that however much you tried to explain your strategy to the other stock traders, they just didn't get it. You wouldn't always pick a winner, but you'd make a lot of money. How confident are you that AI won't be able to find something like that?
  - scotty79 10y ago
    > How about discovering a flaw in human crypto systems? Very good point. As for stock market I'm fairly confident that there won't be strategies that AI can figure out that people won't catch on as fast as any other strategy. I think that's because stock market has very narrow API, and more complex strategies would involve piling up operations, doing multiple steps. Every problem that involves any probabilistic element (or uncertainty) quickly becomes unpredictable as number of steps grows. I don't believe AI can be better at stock market than us because it's a simple game of chance.

scotty79 10y ago

I agree that AI can do marvels in math, but math quickly departs from reality (like string theory) and being able to prove things about some beautiful intricate mathematical structure most of the times doesn't help you much with the world of matter. I recommend Culture novels written by Iain M. Banks. There are vastly powerful AIs but they spend as much time as they can afford in "Infinite Fun Space" which is basically math taken to insane depths. I know that there's a saying that all math is eventually applied but I think this saying is popular because it is sort of paradoxical that some of the most bizarre math eventually found their application. What I think is for math just a bit deeper than what human can grasp, "eventually" quickly becomes larger than the age of universe. About stealing... Hackers are successful not because they are ungodly smart. They are successful because they have the will to look for and exploit vulnerabilities. Intelligence, again, doesn't help all that much. I don't think people who wrote stuxnet were able to do this because they were of superior intelligence. They were just usually intelligent people (which means barely more intelligent than average human) that were sufficiently motivated by huge budget and interesting problems. Super-intelligence won't help you with stock market because it's purely random game. You can see it in results of active management funds. HFT gives impression of algorithms beating people at trading but the thing they use to extract the value is not intelligence, it's speed. If you can trade faster you can beat slower guys because you are playing slightly different random game than they do. If you compare one HFT with other HFT then you are back to random results. So AI won't have upper hand on stock market unless we allow it to trade faster then our non-conscious software. In theory I could image AI persuading people to give it what it wants. Because people are stupid and have flaws that are recurrent and easily exploitable. But again, I don't think intelligence would make such a huge difference. 1000 times more intelligent conmen could be just twice as effective because of chaotic nature of how human flaws interact. > If it's so intelligent, it could design working nanotech. The only reason we don't have it is because it's so hard to engineer. So much complexity and moving parts, It's hard to engineer because it's so damn small, not because it's complex. To build nanotech you'll need to build tools to build tools to build tools to build nanotech. You could do all that but even if you are 1000 times smarter, reality has a speed limit. You can't take a shovel of sand from the beach and build CPU, no matter how intelligent you are. You have to build fab first. > Human brains weren't evolved to be engineers after all, it's a lucky coincidence we are capable of it at all. An AI brain optimized for this task should far exceed humans. The same way chess computers far exceed human chessmasters. Yes. But it can exceed humans at consciousness, charity and compassion even faster because those are purely intellectual things, like chess while engineering is limited by moving atoms and energy around.

makomk 10y ago
The stock market isn't just a purely random game though - it's based on the expected earnings of the companies, so if you can come up with a better estimate than everyone else by combining information better, you can make money. The reason it's hard for humans to make money this way is because everyone else is doing the same thing, so any advantage is short-lived.
- scotty79 10y ago
  Pseuodo-random generators are not purely random - they are based on a seed and an algorithm. For stock market nobody can have a clue what the algorithm is and what meaning the seed has in it. The reason nobody can predict the next value is that at some point something unpredictable happens that can wipe out all your previous gains and some more. Market can remain insane (meaning not behaving according to a model) longer than you can remain solvent.
philh 10y ago
> being able to prove things about some beautiful intricate mathematical structure most of the times doesn't help you much with the world of matter. How about discovering a flaw in human crypto systems? > Super-intelligence won't help you with stock market because it's purely random game. Humans can't reliably exploit the stock market, because we're competing against other humans. It's not that there are no patterns, it's that as soon as someone discovers a pattern, people rush to exploit it and then it's tapped out. Imagine you were the first human stock trader to think of "buy stocks in companies when their product announcements get featured in newspapers"; and imagine that however much you tried to explain your strategy to the other stock traders, they just didn't get it. You wouldn't always pick a winner, but you'd make a lot of money. How confident are you that AI won't be able to find something like that?
- scotty79 10y ago
  > How about discovering a flaw in human crypto systems? Very good point. As for stock market I'm fairly confident that there won't be strategies that AI can figure out that people won't catch on as fast as any other strategy. I think that's because stock market has very narrow API, and more complex strategies would involve piling up operations, doing multiple steps. Every problem that involves any probabilistic element (or uncertainty) quickly becomes unpredictable as number of steps grows. I don't believe AI can be better at stock market than us because it's a simple game of chance.