4 ms·
You can get refurbished Mac Studio M1 Ultra with 128GB VRAM for ~ $3k on ebay. M1 ultra has 800GB/s memory bandwidth, same as the M2 ultra. Not sure if 128GB V
by tarruda 2y ago
You can get refurbished Mac Studio M1 Ultra with 128GB VRAM for ~ $3k on ebay. M1 ultra has 800GB/s memory bandwidth, same as the M2 ultra.
Not sure if 128GB VRAM is enough for running 405b (maybe at 3-bit quant?), but it seems to offer great value for running 70B models at 8-bit.
- a_wild_dandan 2y agoYeah, I would want the 192GB Mac for attempting such hefty models. But I have such basic bitch needs that 405B is overkill haha.
- tarruda 2y ago405b even at low quants would have very low tokens generation speed, so even if you got the 192GB it would probably not be a good experience. I think 405b is the kind of model that only makes sense to run in clusters of A100/H100. IMO it is not worth it, 70b models at q8 are already pretty darn good, and 128gb is more than enough for those.
- a_wild_dandan 2y agoExactly! Have you tried the Phi models? To me, they indicate that we can get much more efficient models. In a few years, 70b on gold standard synthetic data + RL might run circles around SotA. It's such an exciting time to be alive.