4 ms·
That might explain 10-20% utilization, but 0.4% utilization (four tenths of a percent!) should be ringing all sorts of alarm bells. It’s not even that small of
by brrrrrm 4y ago
That might explain 10-20% utilization, but 0.4% utilization (four tenths of a percent!) should be ringing all sorts of alarm bells. It’s not even that small of a model (100x bigger than mobilenet v3)
- touisteur 4y agoThose 300tflops are not exactly easy to use, since they mostly come from tensor cores that are mostly 4x4 fp16 matrix multiply-accumulator into an fp32 accumulator. Not all NN operations fit and TensorRT and cudnn don't always find the nice optimization that will run this thing full speed (it's actually quite hard, mostly cublas and cutlass reach those perfs and even there it's on specific ops / dimensions). Still the rt3090 (?) being a close cousin to the A40 (?) should get 37TFLOPS FP32. Alas, without batching, deep networks are still mostly big pipelines. If author is doing 1 inference per frame and waiting for the inference to finish (so, not masking the pcie back and forth), it's not that surprising... OpenVINO at FP32 is a beast and I'm trying to find an alternative as it's kind of a pain to debug. At least it manages dynamic batch sizes at model compilation step (looking at TVM here)... Haven't tried recent versions on an AVX512_Fp16-supporting proc to check fp16 perf though. Waiting for Keem bay but is it ever coming in pcie or USB factor?