3 ms·
This is where I’m at. While I’d LOVE to have a powerful near-frontier model at home, I don’t have the extra funds necessary to purchase a tricked-out Mac Studio
by NetOpWibby 3mo ago
This is where I’m at. While I’d LOVE to have a powerful near-frontier model at home, I don’t have the extra funds necessary to purchase a tricked-out Mac Studio…and if I did, I’d pay off debts.
- datakan 3mo agoProbably wouldn't matter if you had the funds either since MacStudios are 3-4 months out for purchase.
- netruk44 3mo agoAnd also the weights for GLM 5.2 are 1.5 TB. The only Studios Apple sells today are hard-capped to 64 GB (M4 Max) or 96 GB (M3 Ultra) only. You'd need at least 24 new M4 Max Studios, or 16 new M3 Ultra Studios, or 3 used 512 GB M3 Ultra Studios just to power one GLM 5.2 instance. And even then, you're probably looking at < 5 tokens per second. Personally, I think it makes way more sense to pay a model provider $3/1M tokens.
- m_ke 3mo agoDeep Learning models are designed to get max throughput on GPUs, which ends up being batched workloads. You'll never get proper price competitive utilization on personal hardware vs a cloud inference provider that can batch and pipeline requests optimally to maximize utilization, unless you yourself start running batch jobs. Even once local hardware and models catch up to todays frontiers, by that time there will be 10x better cluster sized models available at a similar discount.
- deleted 3mo ago[deleted]