I think it probably also matters a lot whether or not there's a clear solution. If there's a better than average chance that the optimization strategy is known,

I think it probably also matters a lot whether or not there's a clear solution. If there's a better than average chance that the optimization strategy is known, and will work then it's probably fine to hold off on it. If the way you are writing the program doesn't lend itself to clear solutions for the performance bottlenecks then that's an issue that should be dealt with right away or you risk throwing out a whole lot of work later on.