No. (Well, in the short term as miners leave, the block time increases, which increases the block pressure and increases fees, but the block time re-normalizes

No. (Well, in the short term as miners leave, the block time increases, which increases the block pressure and increases fees, but the block time re-normalizes itself regularly as the mining difficulty adjusts.) The only thing that the number of people mining directly affects is the cost for an attacker to perform a 51% attack on the network.