3 ms·
The same way you use a cloud LLM.
by segmondy 1y ago
The same way you use a cloud LLM.
- oblio 1y agoI think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).
- segmondy 1y agoI know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.
- nomel 1y agoWithout knowing what you were doing with that 1300 lines of code, there's not much insight that can be had from this.
- codazoda 1y agoHow much machine do you have to be able to run Qwen-235B locally?
- drusepth 1y agoAre 235B models classified as local LLMs? I guess they probably are, but others in this thread are probably looking more toward 20B-30B models and sizes that generally fit on the RAM you'd expect in average or slightly-higher-end hardware. My beefy 3D gamedev workstation with a 4090 and 128GB RAM can't even run a 235B model unless it's extremely quantized (and even then, only at like single-digit tokens/minute).
- oblio 1y agoCan you run any of those models without $20 000 worth of hardware that uses as much power and makes as much noise as a small factory?
- segmondy 1y agoI run them with under $3,000 hardware and inference is about 500-600watts with no noise.
- brookst 1y agoI’m a mediocre C programmer on my best day and I assure you a highly competent programmer could probably use 200 lines of code to do what I achieve in 1300. Just counting lines is not a good proxy for how much effort it would take a good programmer. (And I am 100% pro LLM coding, just saying this isn’t a great argument)