You're asking several different questions under a general umbrella which have a lot of specific and long-winded answers, but to be succinct about it: business w

You're asking several different questions under a general umbrella which have a lot of specific and long-winded answers, but to be succinct about it: business want incredible amount of customization in all different ways, and customization is resistant to automation and patterns. Successful frameworks straddle a good balance, which is why they have staying power. I think if you push your question to the extreme, we can ask why Literate Programming never took hold. Wouldn't it be vastly more cost effective for businesses to completely commoditize the entire IT department with just a specific kind of technical writer who could pass along these specs to a service that would compile the application as written? Competitive technology simply isn't that predictable and circumscribed in capability.