3 ms·
Advanced scheduling with K8s... inference requests distributed across a ton of spot/preemptible VMs. We’ve achieved best performance on CPU with MKL. It’s ofte
by ramoz 6y ago
Advanced scheduling with K8s... inference requests distributed across a ton of spot/preemptible VMs. We’ve achieved best performance on CPU with MKL.
It’s often a misconception that you need GPU for inference. It many cases, the overhead of data transfer to GPU makes it much slower than a well tuned CPU.
- DelightOne 6y agoHow do you tune a CPU for inference?
- antoinealb 6y agoYou usually tune the software that runs on it (compiler, libraries, build options etc.). You can also play the game of overclock if you run dedicated hardware.
- DelightOne 6y agoI wondered what those options are. The other comment mention SIMD is interesting. Thanks :)
- deleted 6y ago[deleted]
- 37ef_ced3 6y agoYou can generate custom SIMD inference code for your neural net (tensor shapes, etc.) and CPU cache hierarchy (L1, L2 per thread, etc.) For example, here is ResNet50 for a particular Skylake-X CPU: https://nn-512.com/browse/ResNet50 https://nn-512.com/browse/ResNet50
- DelightOne 6y agoI didn't know this exists, thank you!
- manojlds 6y agoAnd ONNX helps too - https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime
- olavgg 6y agoIntel has also recently announced AVX512 VNNI (Vector Neural Network Instructions), but it is only available for the Xeon CPU family. Not desktop. This should significantly improve inference performance on the cpu.
- 37ef_ced3 6y agoEven AVX512F (Foundation) is powerful: https://NN-512.com https://NN-512.com (It depends on the CPU, how many cores, and the structure of the neural net, but MKL will generally only achieve between 80% and 90% of NN-512's AVX-512 performance)