It is definitely amazing and a huge step forward for RL. However, the paper says that it started off with a policy that beats 84% of players based on imitation

It is definitely amazing and a huge step forward for RL. However, the paper says that it started off with a policy that beats 84% of players based on imitation learning, so it didn't learn this strategy all on its own from scratch. Also, this required hundreds of thousands (millions? some large number) of simulations to learn, it is much more difficult reach that scale of learning in the real world.