Unless P=NP, or we live in Heuristica, any AI superintelligence will hardly be better at solving hard mathematical problems than us. The AI also doesn't know mo

Unless P=NP, or we live in Heuristica, any AI superintelligence will hardly be better at solving hard mathematical problems than us. The AI also doesn't know more science than humanity does, and is limited in its computing power. There are hard limits to what any AGI can do - it cannot break AES-256, invent warp drives, or cure Alzheimer's. it's not an intellectual Santa Claus machine. so Yudkowski's fear of a superintelligence bootstrapping itself into the real world by emailing a DNA sequence to a synthesis company seems absurd. an ASI isn't going to magically be able to design nanotechnology that works on the first try. not to mention that DNA doesn't do anything by itself. His comparison of the 11th century fighting the 21st century is also wrong. 11th century people weren't dumber, they just knew less. This ASI would be smarter, but no know more than we do. Acquiring knowledge and building stuff is the bottleneck.