I always make performance considerations part of my specifications when they matter. Resources are not infinite! Often performances are part of the requirements

I always make performance considerations part of my specifications when they matter. Resources are not infinite! Often performances are part of the requirements even though they are implied. Our requirement for a traffic control system might be primarily concerned with the safety of motorists and pedestrians crossing an intersection. However it's generally implied that this system should eventually allow some cars through and some pedestrians through. And that nobody is left waiting too long! Such a system wouldn't be terribly useful if pedestrians never get a chance to cross because the light never changes. It's not premature optimization to design your system with resource constraints in mind. It's smart engineering! A user shouldn't have input lag to scroll a document. The game engine doesn't need to render an infinite number of entities. An HTTP resource shouldn't be returning every record in the database. If you keep these constraints in mind in the design phase you generally end up with a well performing application. Ignoring them is what I've heard called, "premature pessimization," and is a popular source of performance problems in mature applications.