What you are missing is that in the old days, your buy/sell order was probably handled by a human market maker instead of an HFT's algo, who would push up/down

What you are missing is that in the old days, your buy/sell order was probably handled by a human market maker instead of an HFT's algo, who would push up/down the price much more than 'slightly', hence making you pay more in trading costs. The biggest opponents of HFTs are the major financial institutions who used to provide those market making roles.