I mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from

I mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from scratch on hosting LLMs If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)