Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 qua

Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 quantization saying they get real work done, meanwhile I can't run 4x the size models with BF16 and get them to not eventually fuck up something. Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe. I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.