2 ms·
This is some impressive work. You might like to look at the work HuggingFace has been doing (on non-iOS versions). They can run it in under 1GB RAM: > If is a
by nl 4y ago
This is some impressive work.
You might like to look at the work HuggingFace has been doing (on non-iOS versions). They can run it in under 1GB RAM:
> If is also possible to chain it with attention slicing for minimal memory consumption, running it in as little as < 800mb of GPU vRAM
https://huggingface.co/docs/diffusers/optimization/fp16#offloading-to-cpu-with-accelerate-for-memory-savings https://huggingface.co/docs/diffusers/optimization/fp16#offl...
- liuliu 4y agoCPU offloading doesn't work because Apple has shared memory arch already. The head slicing is similar to https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra... I think it is quite practical only if MPSGraph less mysterious about its allocation strategy. It is not the ideal way though. Ideally, FlashAttention / XFormer is the way to go.