3 ms·
I think there is a money to be made in making huge slow inference chips It needs, * Does the very best model fit in ram * Does it eventually produce an answe
by somanyphotons 2y ago
I think there is a money to be made in making huge slow inference chips
It needs,
* Does the very best model fit in ram
* Does it eventually produce an answer
I'm completely ok with waiting a few minutes for an inference answer if I can avoid spending $250k for an 8x
- mrbungie 2y agoMe too. Just as most problems won't need "phd-level intelligence" (whatever they might want you to think, 10x devs don't require phd skills nor research abilities / test time compute for being good at their job), most problems won't need subsecond latency or response time. I'm fine waiting, not sure if others really need the best now or ever.