5 ms·

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to th

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a high end GPU burning on such task , even if that was actually slightly more expensive. Edit: To clarify I do this partially now on batches of code via openAI api currently. It’s around 1-3 cents for a typical 400-800 line source code file. And I don’t mean feeding the full code base in as a single input.

2bitencryption 4y ago
Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu: On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away. On cpu (i.e. the project llama.cpu), it takes a very, very long time to process the input prompt, before it begins to generate new tokens. Meaning, if you provide a huge copy/paste of code, it will take a long time to ingest all that input, before it begins outputting new tokens. Once it finally starts outputting new tokens, the rate is surprisingly fast, not much slower than gpu. I wish I knew the reason for this, but I'm not an expert :) I've just seen this in practice.
- alchemist1e9 4y ago
  That sounds promising but how about the quality of the output. I’ve been using OpenAI API with chatblade and been giving it code in various languages and it’s quite surprising how well it describes the purpose and code implementation in english. The english description would be useful and relevant for developers trying to quickly familiarize themselves with a code base. For a typical 400-600 line file it looks like it would cost around 1-3 cents per file. However loss of privacy isn’t so great. How do you find output quality of llama + lora for such a task? Code is research code I’d run it on.
- ineedasername 4y ago
  How long is very very long? Am I going to get coffee while it works, going to lunch, doing it right before I go to bed, or hoping it finishes in time to come up with the most perfect epitaph on my tombstone? ;)
- londons_explore 4y ago
  That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.
  - loufe 4y ago
    It is a bug, it's been clearly discussed under the issues of LLaMA.cpp. The wait time is far from atrocious, with a 5600x and the 13B I would wait 15 seconds before input starts for a relatively complex prompt.
- kolinko 4y ago
  Any tips on setting up llama+lora on 30b? There are so many resources that I can't figure out which models to use, and which projects to use to set everything up.
  - nickthegreek 4y ago
    text-generation-webui (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui)
    - kolinko 4y ago
      I tried it, but I can't find the right model for llama+lora 30B . Google, nor Bing ;) is not helpful.
      - nickthegreek 4y ago
        I think this is the one you want: https://huggingface.co/elinas/alpaca-30b-lora-int4 https://huggingface.co/elinas/alpaca-30b-lora-int4
- stolsvik 4y ago
  As I understand these models, that makes no sense whatsoever. Is it the tokenisation that takes this crazy amount of time? Because each additional token should take exactly the same amount of time as the first.
aunty_helen 4y ago
We're a long long long way off that. So check back in two months. Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens. Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.
- alchemist1e9 4y ago
  Are you sure? This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api. I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.
  - aunty_helen 4y ago
    If you can get it within 4k of tokens then your vram needs will be much less. As the other two have pointed out, there's workarounds to this. If you're trying to get codebase level introspection you might need to wait a bit for some of these techs to mature. Exciting space and yea, learning heaps myself day by day.
    - alchemist1e9 4y ago
      The approach I was going to take is generating english code descriptions/summaries in related chunks of under 4k tokens then those can be combined and also summarized. In my opinion the code descriptions gpt 3.5 turbo has been spitting out for me are good quality and concise. I’d argue they are probably better than what many of the developers themselves would write, especially when english isn’t native for the developer.
- 2bitencryption 4y ago
  It's $0.590/hr to rent an A6000 48GB gpu on jarvislabs.ai, which you can do right now, and run the 30b (or 65b with some hard work) model and get incredible results. No confidentiality agreement required :)
mightytravels 4y ago
Same question here - alpaca does not do well with long inputs :( Ideally i can throw a 100 pages PDF at it and get a summary and a response document :)
- alchemist1e9 4y ago
  Can’t you just break it into chunks and the summarize the summaries?
  - mightytravels 4y ago
    Think long lawsuits and discovery. Possible to break it up but really not ideal.
    - alchemist1e9 4y ago
      Yeah I think you just need a tool built with Langchain and it can do what you want.
nathanasmith 4y ago
There is ChatGLM[0], a 6 billion parameter Chinese/English bilingual model that is gaining a reputation as the leading locally runnable LLM for code generation. Maybe look into that. Demo is here[1]. [0]https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md [1]https://huggingface.co/spaces/multimodalart/ChatGLM-6B https://huggingface.co/spaces/multimodalart/ChatGLM-6B
- nmfisher 4y ago
  I tried this the other day for generating basic Chinese conversations - the quality is surprisingly good. While it's still behind the latest GPT, but the gap isn't as large as I thought it would be.
- alchemist1e9 4y ago
  This does seem promising when playing with the demo. That says running on a T4 I believe. This probably means we are only months away from viable self hosted code generation and analysis that could compete with gpt 3.5 turbo. It seems like working on locally runnable LLM that has been fine tuned to focus on code should be a high priority.
garblegarble 4y ago
I'm doing the same tasks but on english text, I have not found a local LLM anywhere near the summarisation ability of GPT 3.5 and 4. That said, there are some summarisation tasks I prefer to run locally even given the massive drop in quality (e.g. using Alpaca), both for privacy reasons and to keep myself up-to-date on local LLMs. P.S. I'm discounting older local summarisation-specific networks, I have found local LLMs to be a jump in quality over them
stolsvik 4y ago
Care to share what prompts you run this with? What is the output you desire, and how do you achieve this?
- alchemist1e9 4y ago
  Prompt: You are an expert python programmer. I will give you some python code below. I want you to describe the code and it’s likely purpose. The answer will be pretty damn good , especially with gpt 4, but it’s still good with gpt 3.5 turbo. Try it out yourself with whatever code you have. This english description of code can then be searched against. That’s the primary practical application I’m thinking about doing currently.

2bitencryption 4y ago

Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu: On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away. On cpu (i.e. the project llama.cpu), it takes a very, very long time to process the input prompt, before it begins to generate new tokens. Meaning, if you provide a huge copy/paste of code, it will take a long time to ingest all that input, before it begins outputting new tokens. Once it finally starts outputting new tokens, the rate is surprisingly fast, not much slower than gpu. I wish I knew the reason for this, but I'm not an expert :) I've just seen this in practice.

alchemist1e9 4y ago
That sounds promising but how about the quality of the output. I’ve been using OpenAI API with chatblade and been giving it code in various languages and it’s quite surprising how well it describes the purpose and code implementation in english. The english description would be useful and relevant for developers trying to quickly familiarize themselves with a code base. For a typical 400-600 line file it looks like it would cost around 1-3 cents per file. However loss of privacy isn’t so great. How do you find output quality of llama + lora for such a task? Code is research code I’d run it on.
ineedasername 4y ago
How long is very very long? Am I going to get coffee while it works, going to lunch, doing it right before I go to bed, or hoping it finishes in time to come up with the most perfect epitaph on my tombstone? ;)
londons_explore 4y ago
That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.
- loufe 4y ago
  It is a bug, it's been clearly discussed under the issues of LLaMA.cpp. The wait time is far from atrocious, with a 5600x and the 13B I would wait 15 seconds before input starts for a relatively complex prompt.
kolinko 4y ago
Any tips on setting up llama+lora on 30b? There are so many resources that I can't figure out which models to use, and which projects to use to set everything up.
- nickthegreek 4y ago
  text-generation-webui (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui)
  - kolinko 4y ago
    I tried it, but I can't find the right model for llama+lora 30B . Google, nor Bing ;) is not helpful.
    - nickthegreek 4y ago
      I think this is the one you want: https://huggingface.co/elinas/alpaca-30b-lora-int4 https://huggingface.co/elinas/alpaca-30b-lora-int4
stolsvik 4y ago
As I understand these models, that makes no sense whatsoever. Is it the tokenisation that takes this crazy amount of time? Because each additional token should take exactly the same amount of time as the first.

aunty_helen 4y ago

We're a long long long way off that. So check back in two months. Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens. Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

alchemist1e9 4y ago
Are you sure? This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api. I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.
- aunty_helen 4y ago
  If you can get it within 4k of tokens then your vram needs will be much less. As the other two have pointed out, there's workarounds to this. If you're trying to get codebase level introspection you might need to wait a bit for some of these techs to mature. Exciting space and yea, learning heaps myself day by day.
  - alchemist1e9 4y ago
    The approach I was going to take is generating english code descriptions/summaries in related chunks of under 4k tokens then those can be combined and also summarized. In my opinion the code descriptions gpt 3.5 turbo has been spitting out for me are good quality and concise. I’d argue they are probably better than what many of the developers themselves would write, especially when english isn’t native for the developer.
2bitencryption 4y ago
It's $0.590/hr to rent an A6000 48GB gpu on jarvislabs.ai, which you can do right now, and run the 30b (or 65b with some hard work) model and get incredible results. No confidentiality agreement required :)

mightytravels 4y ago

Same question here - alpaca does not do well with long inputs :( Ideally i can throw a 100 pages PDF at it and get a summary and a response document :)

alchemist1e9 4y ago
Can’t you just break it into chunks and the summarize the summaries?
- mightytravels 4y ago
  Think long lawsuits and discovery. Possible to break it up but really not ideal.
  - alchemist1e9 4y ago
    Yeah I think you just need a tool built with Langchain and it can do what you want.

nathanasmith 4y ago

There is ChatGLM[0], a 6 billion parameter Chinese/English bilingual model that is gaining a reputation as the leading locally runnable LLM for code generation. Maybe look into that. Demo is here[1]. [0]https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md [1]https://huggingface.co/spaces/multimodalart/ChatGLM-6B https://huggingface.co/spaces/multimodalart/ChatGLM-6B

nmfisher 4y ago
I tried this the other day for generating basic Chinese conversations - the quality is surprisingly good. While it's still behind the latest GPT, but the gap isn't as large as I thought it would be.
alchemist1e9 4y ago
This does seem promising when playing with the demo. That says running on a T4 I believe. This probably means we are only months away from viable self hosted code generation and analysis that could compete with gpt 3.5 turbo. It seems like working on locally runnable LLM that has been fine tuned to focus on code should be a high priority.

garblegarble 4y ago

I'm doing the same tasks but on english text, I have not found a local LLM anywhere near the summarisation ability of GPT 3.5 and 4. That said, there are some summarisation tasks I prefer to run locally even given the massive drop in quality (e.g. using Alpaca), both for privacy reasons and to keep myself up-to-date on local LLMs. P.S. I'm discounting older local summarisation-specific networks, I have found local LLMs to be a jump in quality over them

stolsvik 4y ago

Care to share what prompts you run this with? What is the output you desire, and how do you achieve this?

alchemist1e9 4y ago
Prompt: You are an expert python programmer. I will give you some python code below. I want you to describe the code and it’s likely purpose. The answer will be pretty damn good , especially with gpt 4, but it’s still good with gpt 3.5 turbo. Try it out yourself with whatever code you have. This english description of code can then be searched against. That’s the primary practical application I’m thinking about doing currently.