3 ms·
A Petals dev here. It is not real-time, but we think the speed of ~1 token/sec may be enough for some interactive apps such as chat bots (especially, if you sho
by borzunov 4y ago
A Petals dev here. It is not real-time, but we think the speed of ~1 token/sec may be enough for some interactive apps such as chat bots (especially, if you show tokens to a user once they are generated). You can try one at http://chat.petals.ml http://chat.petals.ml (heads-up: it may be laggy right now due to lots of HN users trying out the system).
Of course, you could do better if you have enough high-end GPUs to host the entire model yourself (3x A100 or 8x 3090). But if you don't, 1 token/sec is much faster than what you get with other existing methods.
- dpflan 4y agoI have not read the technical details, apologies for ignorance, but is there an opportunity for caching?
- jerpint 4y agoProbably not, since you need to compute the activations of unknown inputs and there could be infinitely many variations of them
- KaoruAoiShiho 4y agoWhat are the speeds of other existing methods?
- borzunov 4y agoTheoretical best-case for RAM offloading is 5.5 sec/token, for SSD offloading - 22 sec/token. Implementations we've tested are not faster than 10 sec/token though. See details in our paper: https://arxiv.org/pdf/2209.01188.pdf https://arxiv.org/pdf/2209.01188.pdf