4 ms·

I've got a machine with 4 3090s-- Anyone know which model would perform the best for programming? It's great this can run on a machine w/out a graphics card an

I've got a machine with 4 3090s-- Anyone know which model would perform the best for programming? It's great this can run on a machine w/out a graphics card and is only 3B params, but I have the hardware. Might as well use it.

SushiHippie 3y ago
Here is a leader board of some models https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... Don't know how biased this leaderboard is, but I guess you could just give some of them a try and see for yourself.
- SparkyMcUnicorn 3y ago
  This is a much better leaderboard: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html I've seen the CanAiCode leaderboard several times (and used many of the models listed), but I wouldn't use it to pick a model. It's not a bad list, but the benchmark is too limited. The results are not accurately ranked from best to worst. For example the deepseek 33b model is ranked 5 spots lower than the 6.7b model, but the 33b model is definitely better. WizardCoder 15b is near the top while WizardCoder 33b is ranked 26 spots lower, which is a wildly inaccurate ranking. It's worth noting that those 33b models score in the 70s for HumanEval and HumanEval+ while the 15b model scores in the 50s.
  - SushiHippie 3y ago
    Thanks
tarruda 3y ago
AFAIK deepseek coder family are the best open coding models. I haven't tested, but I think deepseek coder 33b can run in a single RTX 3090 when 4-bit quantized. In your case you might be able to run the non quantized version
Havoc 3y ago
The coding models are all small because speed is crucial. If you need to wait 2 seconds for an autocomplete it becomes near useless.
mistercheph 3y ago
Try mistral 8x7b, which some human evals place above gpt-3.5 and you have enough VRAM and compute to make training a LORA either on your own dataset, or one of the freely available datasets on huggingface worthwhile, or at least interesting
mlboss 3y ago
Did you build a machine with 4x 3090 ? I looking for a way to build such a machine for ML training.
- mesmertech 3y ago
  Wondering the same here
- knicholes 3y ago
  I did! I started by going to vast.ai. I was able to look at the specs of the top-scoring machines. I started with the motherboard (as I knew it could support my 3090s, because some PCIe busses can't handle all that data). Then of course I copied everything else that I could. I ended up using PCIe extenders and zip-tieing (plastic, I should use metal zip ties instead) the cards to a rack I got from Lowes. I'm not too pleased with how it looks, but it works! BTW, depending on where you're at in your ML journey, Jeremy Howard from FastAI says you should focus more on using hosted instances like paperspace until you really need to get your own machine. Unless, of course, you enjoy linux sysadmin tasks. :) It can get really annoying trying to match the right version of CUDA with the version of Pytorch you're trying to get running for the newest model you're trying.
  - knicholes 3y ago
    Here are the parts I can find CPU: https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E16819113642?Item=9SIAD7HJEY2274 https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E168191... Motherboard: https://www.newegg.com/asrock-rack-romed8-2t/p/N82E16813140044?Item=N82E16813140044 https://www.newegg.com/asrock-rack-romed8-2t/p/N82E168131400... SSD: https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796?Item=N82E16820147796 https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796... Computer Power Supply (the GPUs have their own power supplies, run apart from the computer, which I'm told is bad, but... seems to work for years now for me): https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-na-1200w/p/N82E16817139205?Item=N82E16817139205 https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-n... PCIe 4.0 x16 Risers: https://www.newegg.com/p/N82E16812987068?Item=N82E16812987068 https://www.newegg.com/p/N82E16812987068?Item=N82E1681298706... Tower: https://www.newegg.com/black-fractal-design-meshify-2-compact-atx-mid-tower/p/N82E16811352140?Item=N82E16811352140 https://www.newegg.com/black-fractal-design-meshify-2-compac... RAM: https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1X5-003Z-019H7?Item=9SIA7S6DN26040 https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1... Total, without GPUs and their power supplies: $2900

SushiHippie 3y ago

Here is a leader board of some models https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... Don't know how biased this leaderboard is, but I guess you could just give some of them a try and see for yourself.

SparkyMcUnicorn 3y ago
This is a much better leaderboard: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html I've seen the CanAiCode leaderboard several times (and used many of the models listed), but I wouldn't use it to pick a model. It's not a bad list, but the benchmark is too limited. The results are not accurately ranked from best to worst. For example the deepseek 33b model is ranked 5 spots lower than the 6.7b model, but the 33b model is definitely better. WizardCoder 15b is near the top while WizardCoder 33b is ranked 26 spots lower, which is a wildly inaccurate ranking. It's worth noting that those 33b models score in the 70s for HumanEval and HumanEval+ while the 15b model scores in the 50s.
- SushiHippie 3y ago
  Thanks

tarruda 3y ago

AFAIK deepseek coder family are the best open coding models. I haven't tested, but I think deepseek coder 33b can run in a single RTX 3090 when 4-bit quantized. In your case you might be able to run the non quantized version

Havoc 3y ago

The coding models are all small because speed is crucial. If you need to wait 2 seconds for an autocomplete it becomes near useless.

mistercheph 3y ago

Try mistral 8x7b, which some human evals place above gpt-3.5 and you have enough VRAM and compute to make training a LORA either on your own dataset, or one of the freely available datasets on huggingface worthwhile, or at least interesting

mlboss 3y ago

Did you build a machine with 4x 3090 ? I looking for a way to build such a machine for ML training.

mesmertech 3y ago
Wondering the same here
knicholes 3y ago
I did! I started by going to vast.ai. I was able to look at the specs of the top-scoring machines. I started with the motherboard (as I knew it could support my 3090s, because some PCIe busses can't handle all that data). Then of course I copied everything else that I could. I ended up using PCIe extenders and zip-tieing (plastic, I should use metal zip ties instead) the cards to a rack I got from Lowes. I'm not too pleased with how it looks, but it works! BTW, depending on where you're at in your ML journey, Jeremy Howard from FastAI says you should focus more on using hosted instances like paperspace until you really need to get your own machine. Unless, of course, you enjoy linux sysadmin tasks. :) It can get really annoying trying to match the right version of CUDA with the version of Pytorch you're trying to get running for the newest model you're trying.
- knicholes 3y ago
  Here are the parts I can find CPU: https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E16819113642?Item=9SIAD7HJEY2274 https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E168191... Motherboard: https://www.newegg.com/asrock-rack-romed8-2t/p/N82E16813140044?Item=N82E16813140044 https://www.newegg.com/asrock-rack-romed8-2t/p/N82E168131400... SSD: https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796?Item=N82E16820147796 https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796... Computer Power Supply (the GPUs have their own power supplies, run apart from the computer, which I'm told is bad, but... seems to work for years now for me): https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-na-1200w/p/N82E16817139205?Item=N82E16817139205 https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-n... PCIe 4.0 x16 Risers: https://www.newegg.com/p/N82E16812987068?Item=N82E16812987068 https://www.newegg.com/p/N82E16812987068?Item=N82E1681298706... Tower: https://www.newegg.com/black-fractal-design-meshify-2-compact-atx-mid-tower/p/N82E16811352140?Item=N82E16811352140 https://www.newegg.com/black-fractal-design-meshify-2-compac... RAM: https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1X5-003Z-019H7?Item=9SIA7S6DN26040 https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1... Total, without GPUs and their power supplies: $2900