2 ms·
Inference/RAM is 100% the bottleneck, not storage, and it isn't close. These models all have to be fully loaded in memory to run. I think the max RAM on an iPh
by furyofantares 3y ago
Inference/RAM is 100% the bottleneck, not storage, and it isn't close.
These models all have to be fully loaded in memory to run. I think the max RAM on an iPhone is 8GB, max storage is 1TB, so we're talking 2 orders of magnitude more ability to store models vs actually running them.
And of course you'd really rather not use the full RAM for inference.
- BryantD 3y agoYou're 100% correct on the technical challenges here, but the fully loaded requirement is what Apple is trying to address as per the paper they just published. I'm somewhat skeptical that they've solved it since it's a hard problem. We'll see!