optimizations always have to be data-driven, from profiles, starting at the slowest stuff and working your way down, until you satisfy your latency requirements

optimizations always have to be data-driven, from profiles, starting at the slowest stuff and working your way down, until you satisfy your latency requirements unless your slow function materially impacts your performance metrics in a way that you can reproducibly demonstrate, it's a mistake to optimize that code to shave off a few microseconds