29 ms·

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for -

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.

xienze 2y ago
I get where you’re coming from, but the problem with LLMs is that you very regularly need a lot of back-and-forth with them to tease out the information you’re looking for. A more apt analogy might be a coworker that you have to follow up with three or four times, at an hour per. Not so appealing anymore. Doubly so when you have to stand up $2k+ of hardware for the privilege. If I’m paying good money to host something locally, I want decent performance.
- unshavedyak 2y ago
  Agreed. Furthermore, for some tasks like large context code assistant windows i want really fast responses. I've not found a UX i'm happy with yet but for anything i care about i'd want very fast token responses. Small blocks of code which instantly autocomplete, basically.
- MonkeyClub 2y ago
  > If I’m paying good money to host something locally The thing is, however, that at 2k one is not paying good money, one is paying near the least amount possible. TFA specifically is about building a machine on a budget, and as such cuts corners to save costs, e.g. by buying older cards. Just because 2k is not a negligible amount in itself, that doesn't also automatically make it adequate for the purpose. Look for example at the 15k, 25k, and 40k price range tinyboxes: https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox It's like buying a 2k-worth used car, and expecting it to perform as well as a 40k one.
Aurornis 2y ago
> give it a prompt and come back an hour or two later. This is the problem. If your use case is getting a small handful of non-urgent responses per day then it's not a problem. That's not how most people use LLMs, though.

xienze 2y ago

I get where you’re coming from, but the problem with LLMs is that you very regularly need a lot of back-and-forth with them to tease out the information you’re looking for. A more apt analogy might be a coworker that you have to follow up with three or four times, at an hour per. Not so appealing anymore. Doubly so when you have to stand up $2k+ of hardware for the privilege. If I’m paying good money to host something locally, I want decent performance.

unshavedyak 2y ago
Agreed. Furthermore, for some tasks like large context code assistant windows i want really fast responses. I've not found a UX i'm happy with yet but for anything i care about i'd want very fast token responses. Small blocks of code which instantly autocomplete, basically.
MonkeyClub 2y ago
> If I’m paying good money to host something locally The thing is, however, that at 2k one is not paying good money, one is paying near the least amount possible. TFA specifically is about building a machine on a budget, and as such cuts corners to save costs, e.g. by buying older cards. Just because 2k is not a negligible amount in itself, that doesn't also automatically make it adequate for the purpose. Look for example at the 15k, 25k, and 40k price range tinyboxes: https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox It's like buying a 2k-worth used car, and expecting it to perform as well as a 40k one.

Aurornis 2y ago

> give it a prompt and come back an hour or two later. This is the problem. If your use case is getting a small handful of non-urgent responses per day then it's not a problem. That's not how most people use LLMs, though.