I think the argument for using libraries is almost always one of solving for "adequate" performance, not best-case. If you want a library you can actually reuse

I think the argument for using libraries is almost always one of solving for "adequate" performance, not best-case. If you want a library you can actually reuse, you're designing it so that it will work in many situations, and when it doesn't, the design should also support copy-pasting some of the code and reworking it to fit the application. Playing "guess the optimization" adds configuration bulk which can negate the ability of the library to actually specialize in this way. There are specialties, of course, where the point of the library is to be a non-trivial, high-performance implementation. Those are designed towards a specific domain, though.