Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 https://github.com/v

Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune for summarization and knowledge graph tasks: https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Edit: I suppose I actually haven't documented my prompt template usage with the OpenAI API endpoint... There are some special roles that will enable some extra features with this model. Using Langchain (but works with OpenAI API directly): ChatMessagePromptTemplate.from_template(role='meta-task_name', template='summary') ChatMessagePromptTemplate.from_template(role='meta-current_date', template=datetime.now().strftime('%Y-%m-%d')) ChatMessagePromptTemplate.from_template(role='system', template='You are a helpful assistant...') ChatMessagePromptTemplate.from_template(role="user_context", template="Some long document that you want summarized...") ChatMessagePromptTemplate.from_template(role="human", template="Please create a summary of the document") ChatMessagePromptTemplate.from_template(role="ai", template="Okay, first let's gather some key information about the document:") The last "ai" message is optional. The endpoint will complete this message if you include it. You could just not include this and the model will start the response however it wants.