Don't quote me on that, I have really nothing to do with game theory, but as I understand it the paper it shows that reflectiveoracle-computable policies are op

Don't quote me on that, I have really nothing to do with game theory, but as I understand it the paper it shows that reflectiveoracle-computable policies are optimal in the discussed setup (Theorem 25) and that they will yield a Nash equilibrium if all policies are asymptotically optimal (Theorem 28) which is possible because a limit computable reflective oracle exists (Theorem 6) and Thompson sampling is asymptotically optimal. So achieving a Nash equilibrium still requires that all agents play along, they have to have asymptotically optimal policies, if other agents show erratic behavior you are unable to make sense of, you can not respond optimally.