3 ms·
I'm only interested in the local, single user use case. Plus I use a Mac studio for inference, so vLLM is not an option for me.
by tarruda 8mo ago
I'm only interested in the local, single user use case. Plus I use a Mac studio for inference, so vLLM is not an option for me.
- mycall 8mo agoYou can get concurrency gains [0] as local/single user (multi-agent) use case with vLLM with your Mac Studio. [0] https://youtu.be/Ze5XLooTt6g?t=658 https://youtu.be/Ze5XLooTt6g?t=658