Disclosure: I used to work for Google Cloud and know Neural Magic's CEO (Brian Stevens). It's great to see CPU optimizations, but I have a few nits: > As of t

Disclosure: I used to work for Google Cloud and know Neural Magic's CEO (Brian Stevens). It's great to see CPU optimizations, but I have a few nits: > As of this writing, the new A100s do have hardware support for semi-structured sparsity but are not readily available. A100s have been readily available on all major cloud providers for a while. This benchmark would only require one A100, anywhere in the world. Echoing 37ef_ced3's, I would argue that what matters is not raw qps (the A100 will trounce this) but rather qps at a given latency per dollar. The c5.12xlarge is $2.04/hr (and has a sad 12 Gbps NIC...) while the g4dn.2xlarge is is $.752/hr. So the T4 setup is about 20% more qps, but almost 3x better inference/$!