5 ms·
Optimizing LLM Latency
- mofosyne 3y agoWhile not actually speeding things up, I wonder if it's possible to run a very small autocomplete model to guess what the next sentence ahead of time from the LLM on the client side. To at least keep the perceived latency low. Thus if there is a delay, you still get the stream of text coming at a steady rate, but you may have to backtrack and correct for mispredictions.
- gwern 3y agoYes, that sort of speculative execution has long existed for NNs, and has enjoyed a recent resurgence in popularity as people speculate that OA has been using that heavily for its APIs & services. It can work very well for outputs that vary greatly in their predictability (like natural language), but can also backfire - nobody likes seeing the predictions thrash around unpredictably, but if you aren't careful, you accept bad predictions or wind up not saving latency or compute at all.
- luke-stanley 3y agoKeen to see how this compares with Llama.cpp on the same system.
- turnsout 3y agoYeah, seems like a glaring omission?
- qeternity 3y agoArticle discussing LLM performance omits the two most performant libraries: exllama and llama.cpp??
- Tostino 3y agoThey aren't fastest. I get over 1k tokens/sec on llama2 13b on my 2x 3090 with vLLM. They both need to fix their batch inference and it'll be more competitive. They are though.
- freediver 3y agoAny more information how are you running this?
- Tostino 3y agoNothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune for summarization and knowledge graph tasks: https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Edit: I suppose I actually haven't documented my prompt template usage with the OpenAI API endpoint... There are some special roles that will enable some extra features with this model. Using Langchain (but works with OpenAI API directly): ChatMessagePromptTemplate.from_template(role='meta-task_name', template='summary') ChatMessagePromptTemplate.from_template(role='meta-current_date', template=datetime.now().strftime('%Y-%m-%d')) ChatMessagePromptTemplate.from_template(role='system', template='You are a helpful assistant...') ChatMessagePromptTemplate.from_template(role="user_context", template="Some long document that you want summarized...") ChatMessagePromptTemplate.from_template(role="human", template="Please create a summary of the document") ChatMessagePromptTemplate.from_template(role="ai", template="Okay, first let's gather some key information about the document:") The last "ai" message is optional. The endpoint will complete this message if you include it. You could just not include this and the model will start the response however it wants.
- freediver 3y agoThanks, thats awesome. Would you say vLLM on one GPU is as fast or faster than llama.cpp? have you tried mlc-llm?
- Havoc 3y ago> mlc is the fastest Also noticed that mlc+Vulkan has near zero CPU use while Llama+opencl on gpu still end up with a fair bit of cpu use. So seems to be taking a fundamentally different route This was on a rock chip 3588 so not exactly conventional AI hardware