Have you read Developer Hegemony by Erik Dietrich? I wish I could tell you it has the full solution to your problems. I don't think it does, but it's well wort

Have you read Developer Hegemony by Erik Dietrich? I wish I could tell you it has the full solution to your problems. I don't think it does, but it's well worth the read, because it does such a good job at mapping out the problem space that we have. His proposed solution is to form what he calls "efficiencier firms", software consultancies that work from a directive of making the client's business more efficient, rather than working from specs like "this button should be blue". I think that's indeed how software developers should operate, but I don't see how you sell that idea to clients, who like you said, expect someone they can treat like an employee and just tell them to shut up and make the buttons blue. They'd take a very big risk to give you money and wait and see what you come up with. There's also the problem of execution. We have to get a reputation for consistently successful software projects on time and on budget. Which I think is possible given all the tools and literature, but I feel most developers are never get good at it because they are stuck on a few death march projects for their entire careers.