The "gravity well" of the hole is the same whether it's surrounded by vacuum or matter. The only difference is what can possibly fall in. That's exactly it --

The "gravity well" of the hole is the same whether it's surrounded by vacuum or matter. The only difference is what can possibly fall in. That's exactly it -- if the gravity well is a vacuum, then by definition there is nothing to fall into it, and thus nothing to increase the mass, and so the black hole would not grow.