The real answer is that there's a non-zero cost to maintain these pages, and even more so if robots.txt entries and such have to be maintained for them as well.

The real answer is that there's a non-zero cost to maintain these pages, and even more so if robots.txt entries and such have to be maintained for them as well. And if they have no monetary benefit, or even potentially a detriment, it makes more sense for them from a business perspective to just get rid of them. Unfortunately.