7 ms·
YOLOv5 on CPUs: Sparsifying to Achieve GPU-Level Performance
- ThouYS 5y agoI guess pretty soon we'll wonder how anyone was ever ok with using raw non-sparsified models
- ramzyo 5y agoThis is really impressive work. Would be interested to know whether they plan to support ARM CPUs in their runtime engine (currently looks like just a few AMD & Intel CPUs: https://github.com/neuralmagic/deepsparse https://github.com/neuralmagic/deepsparse). Many IoT and embedded devices with application-class processors could benefit from these speedups.
- ddv 5y ago"Is there plan to support sparse NN influence on mobile devices, such as Arm CPUs?" "Based on the amount of work for support, it's on our medium to long-term roadmap currently." https://github.com/neuralmagic/deepsparse/issues/183 https://github.com/neuralmagic/deepsparse/issues/183
- monocasa 5y agoAnd no community pitching in to help with that either. > Open sourcing of the backend runtime is something we're not planning for the near to medium term. It is something we're actively reevaluating, though, as the company and the industry as a whole grows to see what works best for the needs of our users.
- baybal2 5y agoI told you! :) CPUs keep throwing specialised hardware into the garbage bag every time few years after mass adoption of something big, as the computer science improves. People were once telling that real time audio was physically impossible to do on the CPU, now you can do it even on a smartphone CPU.
- monocasa 5y agoSome of that is figuring out the software overhead too. Alexia Massalin was doing audio DSP on her 68k workstation back in the late 80s with her Synthesis kernel. Overall it seems that multiplexing with the right semantics is a majority of the problem to running new workloads on GPCPUs and takes time to understand.
- hrydgard 5y agoCPUs haven't replaced GPUs for graphics though, and are very far away from doing it...
- Cymen 5y agoIt does depend on what you mean by CPUs but the Ryzen G 5000 series are able to play video games at 720p (sometimes 1080p) with the graphics integrated on the CPU.
- sydthrowaway 5y agoErr, isnt that GPU on the same die? That’s the same as Apple Silicon. No one does software rendering do they?
- terafo 5y agoG 5000 series is marginally worse than 5000 series in CPU performance and it is worse than any discrete GPU that was released in last 5 years if we are talking GPU performance.
- 37ef_ced3 5y agoEven without sparsification, AVX-512 CPUs are far more cost-effective for inference. To get your money's worth, you must saturate the processor. It's easy, in practice, with an AVX-512 CPU (e.g., see https://NN-512.com https://NN-512.com or Fabrice Bellard's https://bellard.org/libnc https://bellard.org/libnc), and almost impossible with a GPU. A GPU cloud instance costs $1000+ per month (vs. $10 per month for an AVX-512 CPU). A bargain GPU instance (e.g., Linode) costs $1.50 per hour (and far more on AWS) but an AVX-512 CPU costs maybe $0.02 per hour. That said, GPUs are essential for training. Note that Neural Magic's engine is completely closed-source, and their python library communicates with the Neural Magic servers. If you use their engine, your project survives only as long as Neural Magic's business survives.
- ramzyo 5y ago> Note that Neural Magic's engine is completely closed-source, and their python library communicates with the Neural Magic servers. If you use their engine, your project survives only as long as Neural Magic's business survives. Could be totally off here, but have a feeling this team is going to get scooped up by one of the big co's. Having deja vu of XNOR.ai (purchased by Apple), when partners got the short end of the stick (see Wyze cam saga: https://www.theverge.com/2019/11/27/20985527/wyze-person-detection-ai-startup-canceled-removed-dropped https://www.theverge.com/2019/11/27/20985527/wyze-person-det...)
- shock-value 5y agoGCP lists a T4 which is suitable for inference for between $0.11/hour and $0.35/hour (depending on commitment duration and preemptibility). https://cloud.google.com/compute/gpus-pricing https://cloud.google.com/compute/gpus-pricing
- carbocation 5y agoAgreed - I priced this out for a specific distributed inference task a few months ago and the T4 was cheaper and faster than GPU. On 26 million images using a pytorch model that had 41 million parameters, T4 instances were about 32% cheaper than CPU instances, and took about 45% of the time even after accounting for extra GPU startup time.
- chessgecko 5y agoSuper interesting, but it would probably help the pytorch performance significantly on both the gpu and cpu if they torchscripted the models, it's probably pretty simple given they exported it to onnx and would be more apples to apples It'll be exciting to see what they can do with the a10 gpus when they're available.
- m3at 5y agoAfaik models exported to onnx are already jit compiled in the export process: https://pytorch.org/docs/stable/onnx.html https://pytorch.org/docs/stable/onnx.html
- chessgecko 5y agoI was referring to the benchmark script from the article https://github.com/neuralmagic/deepsparse/blob/main/examples/ultralytics-yolo/benchmark.py https://github.com/neuralmagic/deepsparse/blob/main/examples... It looks like they used torch.load instead of torch.jit.load so I don’t think it’s scripted here, but I didn’t check the model file itself so I’m not entirely sure
- fwsgonzo 5y agoI would love to use this in a "special setting" but it has to be compiled, static, code. So, if there is an implementation for C, C++, Rust or any other language that would be great! I am more than willing to handle the special build myself. I just need a language that compiles to machine code.
- codetrotter 5y agoIs the special setting running it on iOS and distributing it via the Apple App Store? :)
- fwsgonzo 5y agoNo, it's for an edge compute platform that I'm developing for someone. So I will just provide the execution environment, not the actual programs.
- spullara 5y agoI don't know a lot about this but I think you can use this to remove the Python dependency: https://pytorch.org/docs/stable/jit.html https://pytorch.org/docs/stable/jit.html
- m3at 5y agoYou can use apache tvm [1] to get a static runtime. I'm not sure of the state of sparsification acceleration support, but at least quantization is supported. Edit: it's in C++ and has rust bindings [2] [1] https://tvm.apache.org/ https://tvm.apache.org/ [2] https://github.com/apache/tvm/tree/main/rust https://github.com/apache/tvm/tree/main/rust
- deepnotderp 5y agoI'm pretty sure this isn't using the Tensor cores on the GPU. If you see here (https://github.com/ultralytics/yolov5/blob/master/README.md https://github.com/ultralytics/yolov5/blob/master/README.md), the speed of inference on a V100 for YOLOv5s should be 2 ms per image, or 500 imgs/s, not the 44.6 img/s being reported here. This is important as it is more than an order of magnitude off.
- ml_hardware 5y agoMy guess is they are using tensor cores as they report FP16 throughput, but they seem to be measuring at batch size 1, which is hugely unfair to the GPUs. For inference workloads you usually batch incoming requests together and run once on GPU (though this increases latency). A latency/throughput tradeoff curve at different batch sizes would tell the whole story. Also, they are using INT8 on CPU and neglect to measure the same on GPU. All the GPU throughputs would 2x. tl;dr just use GPUs Edit: to the comments below, I agree low-latency can be important to some workloads, but that's exactly why I think we need to see a latency-throughput tradeoff curve. Unfortunately, I'm pretty sure that modern GPUs (A10/A30/A40/A100) basically dominate CPUs even when latency is constrained, and the MLPerf results give a good (fair!) comparison of this: https://developer.nvidia.com/blog/extending-nvidia-performance-leadership-with-mlperf-inference-1-0-results/ https://developer.nvidia.com/blog/extending-nvidia-performan... The GPU throughputs are much, much higher than the CPU ones, and I don't think even NM's software can overcome this gap. Not to mention they degrade the model quality... The last question is whether CPUs are more cost-effective despite being slower, and the answer is still... no. The instances used in this blog post cost: - C5 CPU (c5.12xlarge): $2.04/hr - T4 GPU (g4dn.2xlarge): $0.752/hr NM's best result @ batch-size-1 costs more, at lower throughput, at lower model quality, at ~same latency, than a last-gen GPU operating at half it's capacity. A new A10 GPU using INT8 will widen the perf/$ gap by another ~4x. Also full disclosure I don't work at NVIDIA or anything like that so I'm not trying to shill :) I just like ML hardware a lot and want to help people make fair comparisons.
- bigcorp-slave 5y agoIt depends on your application. If you are running on a smartphone, or on an AR headset, or on a car, or on a camera, etc, you generally do not have the latency budget to wait for multiple frames and run at high batch size.
- c54 5y agoWhat does sparsification mean here?
- throwawaybanjo1 5y agohttps://docs.neuralmagic.com/main/source/getstarted.html#sparsification https://docs.neuralmagic.com/main/source/getstarted.html#spa...
- enricozb 5y agoTangentially related: There is some controversy around YOLOv5's name: https://github.com/ultralytics/yolov5/issues/2 https://github.com/ultralytics/yolov5/issues/2
- sdenton4 5y agoThese days I'm tending to prefer structural sparsity over pruning, because if you make good choices you get high-quality models that are fast on both CPU and (G/T)PU. The EfficientNet architecture (which leans heavily on seperable convolutions) is a good example of this. Using GRUs with block-diagonal matrices and some 'side channel' for communication between the blocks also works very well. A good structurally sparse model can usually be carefully pruned as well, but the gains are a bit smaller once you've already settled into a 'small enough' model architecture.
- boulos 5y agoDisclosure: I used to work for Google Cloud and know Neural Magic's CEO (Brian Stevens). It's great to see CPU optimizations, but I have a few nits: > As of this writing, the new A100s do have hardware support for semi-structured sparsity but are not readily available. A100s have been readily available on all major cloud providers for a while. This benchmark would only require one A100, anywhere in the world. Echoing 37ef_ced3's, I would argue that what matters is not raw qps (the A100 will trounce this) but rather qps at a given latency per dollar. The c5.12xlarge is $2.04/hr (and has a sad 12 Gbps NIC...) while the g4dn.2xlarge is is $.752/hr. So the T4 setup is about 20% more qps, but almost 3x better inference/$!