I tried it using oobabooga's webui side by side with Alpaca 65B loaded in 4 bit on the same AWS instance with 64GB of VRAM. While Alpaca produced 3 tokens/sec,

I tried it using oobabooga's webui side by side with Alpaca 65B loaded in 4 bit on the same AWS instance with 64GB of VRAM. While Alpaca produced 3 tokens/sec, Falcon produced 0.17 tokens/sec. So it is very slow with the current tooling still.