What is stopping people from setting up multiple pools and getting 50% that way? How do you really know all those pool operators are actually different people?

What is stopping people from setting up multiple pools and getting 50% that way? How do you really know all those pool operators are actually different people? For a 'trustless' system, it seems like you need to place an awful lot of to trust in this small, secretive cabal of mining operators.