4 ms·
Or, do your inference using an AVX-512 CPU: https://NN-512.com https://NN-512.com (open source, free software, no dependencies) With batch size 1, NN-512 is e
by 37ef_ced3 5y ago
Or, do your inference using an AVX-512 CPU:
https://NN-512.com https://NN-512.com (open source, free software, no dependencies)
With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider.
Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster.
If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
If you need Transformers instead of convolutions, Fabrice Bellard's LibNC is a good solution: https://bellard.org/libnc/ https://bellard.org/libnc/
- theo31 5y agoOh that's very interesting, how ready for production is it? It only works for TF right? > If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year. Definitely the cheapest way. We've been in business for more than a year already actually :)
- 37ef_ced3 5y agoNN-512 has no connection to TensorFlow. It is an open source Go program (with no dependencies) that generates C code (with no dependencies). And it's fully ready for production. Similarly, LibNC is stand-alone, and Fabrice Bellard (author of FFmpeg, QEMU, etc.) will release the source to anyone who asks for it. I'm giving performance comparisons versus TensorFlow, which I consider to be a standard tool. People who use your proprietary, closed, black-box service are dependent on the well-being of your business. You could vanish tomorrow.