Don't get me wrong -- performance can matter, but in the end it's up to the user of your cryptographic solution/process/abstraction to decide what tradeoff is a

Don't get me wrong -- performance can matter, but in the end it's up to the user of your cryptographic solution/process/abstraction to decide what tradeoff is acceptable to them. If you're responsible about it, you try to explain to potential users what the tradeoffs are -- you don't try to optimize before you can be sure that you're doing so sensibly (see next paragraph). Of course you should try to optimize as much as is sensible, but if you don't have any process in place (unit tests, integration tests, fuzz testing, formal methods, timing attacks, whatever you can throw at it) that lend some assurance that Implementation#1 (optimized) actually does exactly the same as Implementation#0 (less or unoptimized) under all circumstances, then you should be as conservative as humanly possible... and choose to NOT optimize unless/before you have those processes in place. (I'll happily admit that this is somewhat armchairy -- I'm glad I don't work in a particularly security sensitive area. I'm not sure I could handle all the obligations/pressures myself.)