3 ms·
I think there are a couple of basic questions need answered before we can find a good solution: 1) What are you trying to do? 2) What's your budget? Generica
by MPSimmons 3y ago
I think there are a couple of basic questions need answered before we can find a good solution:
1) What are you trying to do?
2) What's your budget?
Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question.
When I was faced with this question, I bought the cheapest 4060 Ti with 16GB I could find. It does "okay". Here's an example run:
Llama.generate: prefix-match hit
llama_print_timings: load time = 627.53 ms
llama_print_timings: sample time = 415.30 ms / 200 runs ( 2.08 ms per token, 481.58 tokens per second)
llama_print_timings: prompt eval time = 162.12 ms / 62 tokens ( 2.61 ms per token, 382.44 tokens per second)
llama_print_timings: eval time = 8587.32 ms / 199 runs ( 43.15 ms per token, 23.17 tokens per second)
llama_print_timings: total time = 9498.89 ms
Output generated in 9.79 seconds (20.43 tokens/s, 200 tokens, context 63, seed 1836128893)
I'm using the text-generation-webui to provide the OpenAI API interface. It's pretty easy to hit:
import os
import openai
url = "http://localhost:7860/v1"
openai_api_key = os.environ.get("OPENAI_API_KEY")
client = openai.OpenAI(base_url=url, api_key=openai_api_key)
result = client.chat.completions.create(
model="wizardlm_wizardcoder-python-13b-v1.0",
messages = [
{"role":"system", "content":"You are a helpful AI agent. You are honest and truthful"},
{"role":"user", "content": "What is the best approach when writing recursive functions?"},
]
print(result)
But again, it just depends on what you want to do.
- godelski 3y ago> What are you trying to do? This is by far the most important question. Because frankly, I can run LLaMA on my raspberry pi. It's slow as hell and not suited for any real time task but there are definitely operations where this would be an appropriate cost effective solution (preferably with actually a smaller distilled model). There is no one size-fits all solution. The general advice is going to be a general mid tier graphics card but I assume that's information OP already has or could have found just as easily by typing this question into Google or any LLM. So if you (OP) want better advice, we got to have more information. The more detailed, the better (if this is a commercial application, then the answer is A100 because geforce cards are not allowed to be used for commercial environments, but no one's really going to stop you either). Ask vague question, get vague answers. But we will ask refining questions to help you ask better questions too :)
- Closi 3y agoAnd a third question - what kind of machine are you looking for? If you want a laptop, that's going to be a very different machine to a desktop.
- m463 3y agoI've noticed that llama 2 + llama.cpp doesn't seem to even use the GPU much. I tried a better gpu (more speed, more memory) and my inference speed didn't increase.