The problem is that every new software project requires defining and solving a brand-new problem, and that "communication and setting expectations" part is real

The problem is that every new software project requires defining and solving a brand-new problem, and that "communication and setting expectations" part is really the problem of, first, specifying the problem and then, second, guessing how long it will take to solve the problem. Insert the old joke about "multiply by three and add five". Within corporate environments, the ultimate problem is that the people who allocate the resources are usually the least knowledgeable about the costs of "digging a hole" (i.e. technical debt). Of course, there is a great deal of blame to be placed on our -- the developers' -- shoulders, but guesstimating software delivery is difficult enough without non-technical people setting ridiculous timelines. I'd say the solution to the problem is to have managers select competent, driven developers who care about quality, and then for those managers to trust the technical teams to get the job done right. For such a team to work, each piece must work well and work hard.