3 ms·

Article discussing LLM performance omits the two most performant libraries: exllama and llama.cpp??

Tostino 3y ago
They aren't fastest. I get over 1k tokens/sec on llama2 13b on my 2x 3090 with vLLM. They both need to fix their batch inference and it'll be more competitive. They are though.
- freediver 3y ago
  Any more information how are you running this?
  - Tostino 3y ago
    Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune for summarization and knowledge graph tasks: https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Edit: I suppose I actually haven't documented my prompt template usage with the OpenAI API endpoint... There are some special roles that will enable some extra features with this model. Using Langchain (but works with OpenAI API directly): ChatMessagePromptTemplate.from_template(role='meta-task_name', template='summary') ChatMessagePromptTemplate.from_template(role='meta-current_date', template=datetime.now().strftime('%Y-%m-%d')) ChatMessagePromptTemplate.from_template(role='system', template='You are a helpful assistant...') ChatMessagePromptTemplate.from_template(role="user_context", template="Some long document that you want summarized...") ChatMessagePromptTemplate.from_template(role="human", template="Please create a summary of the document") ChatMessagePromptTemplate.from_template(role="ai", template="Okay, first let's gather some key information about the document:") The last "ai" message is optional. The endpoint will complete this message if you include it. You could just not include this and the model will start the response however it wants.
    - freediver 3y ago
      Thanks, thats awesome. Would you say vLLM on one GPU is as fast or faster than llama.cpp? have you tried mlc-llm?
      - Tostino 3y ago
        Llama.cpp has faster single thread inference, but I actually haven't tried their batched inference yet, so I don't know what throughput I can get through it.

They aren't fastest. I get over 1k tokens/sec on llama2 13b on my 2x 3090 with vLLM. They both need to fix their batch inference and it'll be more competitive. They are though.

freediver 3y ago
Any more information how are you running this?
- Tostino 3y ago
  Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune for summarization and knowledge graph tasks: https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Edit: I suppose I actually haven't documented my prompt template usage with the OpenAI API endpoint... There are some special roles that will enable some extra features with this model. Using Langchain (but works with OpenAI API directly): ChatMessagePromptTemplate.from_template(role='meta-task_name', template='summary') ChatMessagePromptTemplate.from_template(role='meta-current_date', template=datetime.now().strftime('%Y-%m-%d')) ChatMessagePromptTemplate.from_template(role='system', template='You are a helpful assistant...') ChatMessagePromptTemplate.from_template(role="user_context", template="Some long document that you want summarized...") ChatMessagePromptTemplate.from_template(role="human", template="Please create a summary of the document") ChatMessagePromptTemplate.from_template(role="ai", template="Okay, first let's gather some key information about the document:") The last "ai" message is optional. The endpoint will complete this message if you include it. You could just not include this and the model will start the response however it wants.
  - freediver 3y ago
    Thanks, thats awesome. Would you say vLLM on one GPU is as fast or faster than llama.cpp? have you tried mlc-llm?
    - Tostino 3y ago
      Llama.cpp has faster single thread inference, but I actually haven't tried their batched inference yet, so I don't know what throughput I can get through it.