3 ms·

We're a long long long way off that. So check back in two months. Jokes aside, the limiting factor will be either a technique to pack all of the code into smal

We're a long long long way off that. So check back in two months. Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens. Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

alchemist1e9 4y ago
Are you sure? This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api. I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.
- aunty_helen 4y ago
  If you can get it within 4k of tokens then your vram needs will be much less. As the other two have pointed out, there's workarounds to this. If you're trying to get codebase level introspection you might need to wait a bit for some of these techs to mature. Exciting space and yea, learning heaps myself day by day.
  - alchemist1e9 4y ago
    The approach I was going to take is generating english code descriptions/summaries in related chunks of under 4k tokens then those can be combined and also summarized. In my opinion the code descriptions gpt 3.5 turbo has been spitting out for me are good quality and concise. I’d argue they are probably better than what many of the developers themselves would write, especially when english isn’t native for the developer.
2bitencryption 4y ago
It's $0.590/hr to rent an A6000 48GB gpu on jarvislabs.ai, which you can do right now, and run the 30b (or 65b with some hard work) model and get incredible results. No confidentiality agreement required :)
- Casteil 4y ago
  Do you know how their "spot pricing" works, exactly? "lets you use spare GPUs" doesn't tell us much. Is it granular to how long the duration the model is actively processing your prompt? ..or do you get dinged for an hour immediately and again on the hour?
barking_biscuit 4y ago
>Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k. I haven't started to play with LLMs locally in anger yet, but I was under the impression that you could use a 4090 in combination with FlexGen to achieve this rather than having to buy special hardware?
- tetete 4y ago
  With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft https://huggingface.co/blog/trl-peft But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.
  - nullsense 4y ago
    Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...
    - all2 4y ago
      Could you package all that compute into a "virtual" graphics card?
pmoriarty 4y ago
"Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k." Does anyone have a projection based on historical GPU cost reductions as to how long we have to wait until a DGX unit costs as little as a 4090 does today?
- phh 4y ago
  The issue is largely around available RAM capacity, and sadly RAM price have barely decreased in the last 5-10 years

alchemist1e9 4y ago

Are you sure? This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api. I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.

aunty_helen 4y ago
If you can get it within 4k of tokens then your vram needs will be much less. As the other two have pointed out, there's workarounds to this. If you're trying to get codebase level introspection you might need to wait a bit for some of these techs to mature. Exciting space and yea, learning heaps myself day by day.
- alchemist1e9 4y ago
  The approach I was going to take is generating english code descriptions/summaries in related chunks of under 4k tokens then those can be combined and also summarized. In my opinion the code descriptions gpt 3.5 turbo has been spitting out for me are good quality and concise. I’d argue they are probably better than what many of the developers themselves would write, especially when english isn’t native for the developer.

2bitencryption 4y ago

It's $0.590/hr to rent an A6000 48GB gpu on jarvislabs.ai, which you can do right now, and run the 30b (or 65b with some hard work) model and get incredible results. No confidentiality agreement required :)

Casteil 4y ago
Do you know how their "spot pricing" works, exactly? "lets you use spare GPUs" doesn't tell us much. Is it granular to how long the duration the model is actively processing your prompt? ..or do you get dinged for an hour immediately and again on the hour?

barking_biscuit 4y ago

>Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k. I haven't started to play with LLMs locally in anger yet, but I was under the impression that you could use a 4090 in combination with FlexGen to achieve this rather than having to buy special hardware?

tetete 4y ago
With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft https://huggingface.co/blog/trl-peft But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.
- nullsense 4y ago
  Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...
  - all2 4y ago
    Could you package all that compute into a "virtual" graphics card?

pmoriarty 4y ago

"Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k." Does anyone have a projection based on historical GPU cost reductions as to how long we have to wait until a DGX unit costs as little as a 4090 does today?

phh 4y ago
The issue is largely around available RAM capacity, and sadly RAM price have barely decreased in the last 5-10 years