3 ms·
You only need 64 gb of cpu ram to run gpt-oss, or one h100.
by fzzzy 1y ago
You only need 64 gb of cpu ram to run gpt-oss, or one h100.
- claytonjy 1y agoyou can’t really buy H100s except in multiples of 8. If you want fewer, you must rent. Even then, hyperscalers tend to be a bit inflexible there; GCP only recently added support for smaller shapes, and they can’t yet be reserved, only on-demand or spot iirc.
- janice1999 1y agoI assume you're talking that's a quantised 20B model on a several thousand dollar Mac? That's really impressive and huge progress but is that indicative of companies serving thousands of users? They still have to buy Nvidia at the end of the day.