The problem with 8bit at the moment is massive performance degradation with bitsandbytes. Recent improvements in 4bit inference mean that 8bit is now a massive

The problem with 8bit at the moment is massive performance degradation with bitsandbytes. Recent improvements in 4bit inference mean that 8bit is now a massive laggard (although there’s no reason not to expect this to resolve).