4 ms·

I will only want to use this if i'm able to host the LLM locally. Currently using CodeGPT and running CodeBooga 34b @ 4bit on my work laptop. This is good enou

I will only want to use this if i'm able to host the LLM locally. Currently using CodeGPT and running CodeBooga 34b @ 4bit on my work laptop. This is good enough for me, but i'd love some more advanced functionality, such as code completion. CodeGPT - Repo: https://github.com/carlrobertoh/CodeGPT https://github.com/carlrobertoh/CodeGPT - Plugin: https://plugins.jetbrains.com/plugin/21056-codegpt https://plugins.jetbrains.com/plugin/21056-codegpt

janice1999 3y ago
> running CodeBooga 34b @ 4bit on my work laptop. How much RAM or VRAM does that require?
- valine 3y ago
  It’ll be around 17GB + context. For longer contexts you’ll add a couple GBs.
- _boffin_ 3y ago
  Max ram is saying 22.72GB. They gave me a laptop with a M2 MAX and 64GB of ram. I'm loving the fact that i can toss any piece of code in there without having to worry about data privacy issues and or any retention policies of service providers.
- rf15 3y ago
  In my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you'd need for an 8 bit model. with a 4-bit quantization that's halved, so your ball park is 17 GB plus a few GB for context. You can run these in RAM, but it's not very fast with (if I had to guess) slightly less than 1 Token/sec. on most non-apple consumer devices.
qoxorr 3y ago
[dead]
adamjc 3y ago
Isn't that going to be extremely slow? I can only realistically run 7B 5-bit models on my RTX 3060, anything more and it offloads to the CPU. My responses go from almost-instantaneous to 3mins+.
- _boffin_ 3y ago
  It seems like it's running at comparable speeds to GPT4 prior to Turbo. I could be wrong, but what I'm trying to say, it ain't bad at all.
- bdavbdav 3y ago
  This is where the Mac world shines.
  - lamtung 3y ago
    would a 32gb M2 Max be able to run a 34b-model?
rf15 3y ago
Give it another year. In theory it's not going to be difficult, but sampling speed/power consumption is prohibitive for normal hardware at the moment.
- ilc 3y ago
  Honestly, with a 3090 costing like $700 used. It is within reach of many on this site quite easily, assuming they have a machine to plug the 3090 into. People seem to think the hardware for this stuff is way out there... it is getting to be within reach VERY quickly.
  - rf15 3y ago
    Yes, but if you're looking for code completion at the quality level of Codex/Github Copilot, that's still a bit off.
    - _boffin_ 3y ago
      Can you show me the metrics for that, please. Additionally, what's your company's IP policy? Do they let you send their code to a remote company or do they see that as a breach of security?
      - dzhiurgis 3y ago
        I'm sure there are all sort of companies with very strict isolation requirements. But your typical enterprise will run your code editor in cloud VDI, host on atlassian, deploy using atlassian, deploy on some other cloud provider and be developed by double outsourced junior developer who's got warezd windows licence. I fail to see how adding another party who says they explicitly don't collect your data will make any difference.
    - ilc 3y ago
      Don't fall into the trap of: It has to be the best or it is useless. The question for me is: Does it help me get my job done. Does it assist me in ways I find useful.
ilc 3y ago
Been using CodeGPT a while myself. Great tool, send e-mail to the author, seems like a good person. I actually hook it up to my Windows gaming rig. The 4090 gets the job done, when I'm not gaming. (I just start the server when I go work, and then stop it when I want to game. It isn't that big a deal. I haven't hit the use-case where I want to use the AI to develop game mods yet. But I'll cross that bridge when I get there. ;) )
- _boffin_ 3y ago
  Which model are you using? I tried the deepseek 34b or whatever, but it goes nuts at times. Also, what params do you have for Prompt Context Size and Max Completion Tokens?
  - ilc 3y ago
    Currently Phind_Phind-CodeLlama-34B-v2, gonna try CodeBooga because, I can... that's the point. Use what ya like. 4k/32k, if I remember right. An interesting thing to do is actually try to use a true coding model in a chat. I had one model puking out its stack overflow training data, down to including user IDs. Oops.

janice1999 3y ago

> running CodeBooga 34b @ 4bit on my work laptop. How much RAM or VRAM does that require?

valine 3y ago
It’ll be around 17GB + context. For longer contexts you’ll add a couple GBs.
_boffin_ 3y ago
Max ram is saying 22.72GB. They gave me a laptop with a M2 MAX and 64GB of ram. I'm loving the fact that i can toss any piece of code in there without having to worry about data privacy issues and or any retention policies of service providers.
rf15 3y ago
In my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you'd need for an 8 bit model. with a 4-bit quantization that's halved, so your ball park is 17 GB plus a few GB for context. You can run these in RAM, but it's not very fast with (if I had to guess) slightly less than 1 Token/sec. on most non-apple consumer devices.

qoxorr 3y ago

[dead]

adamjc 3y ago

Isn't that going to be extremely slow? I can only realistically run 7B 5-bit models on my RTX 3060, anything more and it offloads to the CPU. My responses go from almost-instantaneous to 3mins+.

_boffin_ 3y ago
It seems like it's running at comparable speeds to GPT4 prior to Turbo. I could be wrong, but what I'm trying to say, it ain't bad at all.
bdavbdav 3y ago
This is where the Mac world shines.
- lamtung 3y ago
  would a 32gb M2 Max be able to run a 34b-model?

rf15 3y ago

Give it another year. In theory it's not going to be difficult, but sampling speed/power consumption is prohibitive for normal hardware at the moment.

ilc 3y ago
Honestly, with a 3090 costing like $700 used. It is within reach of many on this site quite easily, assuming they have a machine to plug the 3090 into. People seem to think the hardware for this stuff is way out there... it is getting to be within reach VERY quickly.
- rf15 3y ago
  Yes, but if you're looking for code completion at the quality level of Codex/Github Copilot, that's still a bit off.
  - _boffin_ 3y ago
    Can you show me the metrics for that, please. Additionally, what's your company's IP policy? Do they let you send their code to a remote company or do they see that as a breach of security?
    - dzhiurgis 3y ago
      I'm sure there are all sort of companies with very strict isolation requirements. But your typical enterprise will run your code editor in cloud VDI, host on atlassian, deploy using atlassian, deploy on some other cloud provider and be developed by double outsourced junior developer who's got warezd windows licence. I fail to see how adding another party who says they explicitly don't collect your data will make any difference.
  - ilc 3y ago
    Don't fall into the trap of: It has to be the best or it is useless. The question for me is: Does it help me get my job done. Does it assist me in ways I find useful.

ilc 3y ago

Been using CodeGPT a while myself. Great tool, send e-mail to the author, seems like a good person. I actually hook it up to my Windows gaming rig. The 4090 gets the job done, when I'm not gaming. (I just start the server when I go work, and then stop it when I want to game. It isn't that big a deal. I haven't hit the use-case where I want to use the AI to develop game mods yet. But I'll cross that bridge when I get there. ;) )

_boffin_ 3y ago
Which model are you using? I tried the deepseek 34b or whatever, but it goes nuts at times. Also, what params do you have for Prompt Context Size and Max Completion Tokens?
- ilc 3y ago
  Currently Phind_Phind-CodeLlama-34B-v2, gonna try CodeBooga because, I can... that's the point. Use what ya like. 4k/32k, if I remember right. An interesting thing to do is actually try to use a true coding model in a chat. I had one model puking out its stack overflow training data, down to including user IDs. Oops.