3 ms·
There is some recent work to make the model loading significantly faster by InferX. They are claiming to be able to load the a 7b in under 2 seconds. I haven’t
by mountainriver 1y ago
There is some recent work to make the model loading significantly faster by InferX. They are claiming to be able to load the a 7b in under 2 seconds.
I haven’t tried it but if that’s the case it’s a game changer
- charles_irl 1y agoWe've talked to them and there's some impressive technology there!