Optimized TF serving would perform similarly to Panini however, it's really hard to find good documentation on optimizing TF serving compilation parameters. Pan

Optimized TF serving would perform similarly to Panini however, it's really hard to find good documentation on optimizing TF serving compilation parameters. Panini automatically finds the right batch size to maximize the throughput and it adaptively changes. We also have a technique to reduce bound tail latency. I would love it for you to try it and provide me some feedback. Thanks