We have tested this code on a 16GB Nvidia T4 GPU. The entire inference process uses less than 4GB GPU memory. You don't need 16GB of VRAM for this, it's just w

We have tested this code on a 16GB Nvidia T4 GPU. The entire inference process uses less than 4GB GPU memory. You don't need 16GB of VRAM for this, it's just what they had available for testing.