5 ms·

The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is rea

by dofm 4mo ago

The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. I don't know how much serious hands-free agentic coding I will ever do on my MacBook alone, but I do know that I would not have got so far into understanding this without tinkering with local models, llama.cpp, LM Studio, and LM Studio and all that. I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus. Until, that is, I could poke around with setting it up on my own (secondhand) machine, watching the API calls, understanding some of the terminology. I didn't even buy the machine for that; it's just adequate to the task. The Neo is too small to really get much benefit from this opportunity to make it more visceral and knowable.

rusk 4mo ago
> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. I found LM studio to be a nice starting point. Frindlier and more featureful than Ollama and not as intimidating as llama.cpp (though you will want to use that eventually)
- dofm 4mo ago
  LM Studio is also nice because of the way the interface explains things; parameters have explanations and hints. It has been designed by people who really care about making it understandable. I tried Ollama but I've settled on Unsloth Studio generally; once things really settle down I'll just run the llama-server UI, which is pretty nice. A friend is tinkering with LLMs for amusement on a 16GB Raspberry Pi 5, and when I explained that llama.cpp now had a typical web chat interface he was so happy — it's amazing what the "table stakes" are now.
ddalex 4mo ago
I just got Claude to download and install all the models and servers and agents and prepare all the launch scripts for me... no need to learn, just ask it to do it for you
- kdkdjduxnd 4mo ago
  [dead]
- dofm 4mo ago
  Right, but I am a middle-aged bloke who is experiencing existential angst about whether I can carry on in this industry. I have a pretty deep, maybe paranoid need to be confident I have an intrinsic understanding, and I have found in my life that lessons come to you when you make yourself open to learning. So I need to build on top of what I know, taking as much of the hard way as I can bear to take at any one time — it has to be not quite difficult enough to put me off. I can't really explain what I have learned this way that is different, but I feel it in a way that I wouldn't if I'd simply pushed a button. For the same reason, I have a really basic 3D printer that I've set up myself, set up Klipper, configured how I want it, learned how to calibrate, all that. And now I can say that I feel I have an understanding of 3D printing. I could hold my head above water in a discussion with a real expert, maybe find work in an adjacent field where my insights would keep me grounded. I can afford a really good printer that has all that set up, and more, has no problems. But I'd just be someone who has a 3D printer. (Also who am I kidding about the existence of a printer with no problems)
  - greyskull 4mo ago
    This really resonates with me, and I'm only a decade and change into my career. I use claude a lot day to day. I try to use it sensibly, making me more productive and produce better work. I'm also trying not to lose understanding along the way. I want to be able to actually talk to the conclusions I'm reaching. I have colleagues that seem perfectly content to delegate too much to the agents, and it saddens me. It feels like there will be swaths of engineers that didn't train some of the critical thinking skills that I take for granted. I certainly see it in slack discourse around anything more complicated than a feature implementation. Maybe I'm just cynical. Time will tell, I suppose.
    - sanderjd 4mo ago
      For me (about halfway between you and dofm in my career by your own statements in this thread), it's a dream at the moment. I can delegate all the tedious stuff that I've done "the hard way" a thousand times already and feel I have very little of value remaining to learn, so that I can spend more time on all the things that are actually new and thus much more interesting.
cyanydeez 4mo ago
I've setup to local paradigms for local coding: - opencode with it's webui - deer-flow with it's research/powered front end They both run websites so you don't have to baby sit them (eg, keep your mac open). I've build a pdf compressor over a few days by first having deer flow try and research the frameworks and pipeline. It stalls out because its not really a fluid programmer. Once it stalls out, I transferred it (manually for now) to opencode and it's refactoring it because it's just a collective bundle of sticks and it needs a lot of testing to tweak out the limited scop context. LLMs can't really hold large scopes (locally anyway, from what I've read from HN, it's possible with longer context). It'll complete in a few days with maybe 3-4 hours of full attention interaction, but it's running 3x that without my attention. Obviously, if I paid more attention it'd run quicker, but since it's local, it's not pumping out large volumes of code, it's mostly looping over tests and capabilities as observed. It's running Qwen3.6 35B MoE on a AMD 128GB strix halo. If I switched to the dense models, perhaps it'd be smarter, but the trade off seems to be much slower gen.
- dofm 4mo ago
  > - opencode with it's webui Have you tried Paseo? I have opencode in a VM, and the paseo daemon running in the VM, and then the Paseo Mac app. Really nice. (You can also use the Opencode GUI to frame a remote opencode web interface)
  - c-hendricks 4mo ago
    You can also just add OpenCode web as a PWA, if that's what you mean by "frame". I'm gonna check out paseo, but am not looking forward to all the ram the agent needs + all the ram paseo needs
    - c-hendricks 4mo ago
      Have checked out Paseo, not sure what it offers over opencode web though. Definitely seems great if you're using other harnesses, but it seems like all it has over opencode web is split views and native apps. Neither of those really matter to me, plus you lose some opencode goodies. The preview urls are a neat idea, but our dev servers at work are mostly port independent and required to be on a certain subdomain for auth.
pizza234 4mo ago
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem. Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are currently an expensive toy. When I try to program with a local LLM (I'm on a 32/128 GB system), I end up wasting time compared to a cloud LLM.
- dofm 4mo ago
  Again, I would not argue against any of this. And I can't say that I won't switch to openrouter (even just for the same models) at some point. But one of the things I have found about my own process learning is that some lessons only come to you when you make yourself available to them. And if that means doing things the difficult way, that is what you should do.
  - wahnfrieden 4mo ago
    Difficult... and wastefully expensive
    - dofm 4mo ago
      I mean, it's a (secondhand) computer I bought for other tasks (processing very large photos, compiling large apps quickly). It's running all the time. It can also run LLMs when I want to. The rest of my life is ultra-frugal so I am relaxed about this.
      - _puk 4mo ago
        Don't bite. You're right. Having spent a good weekend learning how to perform latent-steering through playing with pytorch and a local Gemma4 model, there is no way I could have groked any of that in the the way I did without hands on time. This is on an M3 Max 36GB I've had for a couple of years. No further outlay needed.
codazoda 4mo ago
I agree with the learning aspect, but I have another motivation. I suspect that closed models might become too expensive to run for personal hobbyist use. I’ve been planning to buy a 64GB machine just to allow the limited local models this enables.
oceanplexian 4mo ago
Honestly your best bet is to buy a $20 Claude subscription, ask Claude to set it all up with Pi and llama.cpp and come back in 20 minutes after a cup of coffee. This is also a good idea because it will help set expectations of what a local model can do vs. a frontier model.
- mullen 4mo ago
  This is what I did after struggling to get llama.cpp working at a decent speed on my M1 Macbook. The secret is to very specific with your needs and targeted in what you are using llama.cpp for. Mine setup is just about strictly for qwen3-coder and now, I get a fairly decent speed out of it. I also installed Cursor to check Claude and it all worked out well.
  - trey-jones 4mo ago
    I don't know why we're even talking about Qwen3.6 for writing code when qwen3-coder exists. My experience is there's no contest. I'm using 30b with 96k context on a dedicated server.
    - fouc 4mo ago
      For agentic workflows like tool use, editing codebases, multi-turn debugging?
  - kristianp 4mo ago
    Are you talking about Qwen3 Coder 30b a3b Instruct from August 2025, which is a non-reasoning model? Or the more recent "Qwen3 Coder Next" from Feb this year with 80b params, 3b active? I found Qwen3 coder next to be quite good on openrouter [1], but couldn't run it locally. [1] https://openrouter.ai/qwen/qwen3-coder-next https://openrouter.ai/qwen/qwen3-coder-next
ricardobayes 4mo ago
I'd say give it some time for the dust to settle. This field badly needs standardized benchmarks even before the conversation around model goodness can start.
not_kurt_godel 4mo ago
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Agree having a powerful machine is really worth it in general for professionals, but strong disagree that running local LLMs has anything to do with it. It's hard enough as it is getting a good ROI on your time/money prompting/wrangling with frontier models. IMO leaning on the comparatively limited capabilities of local LLMs is best avoided in favor of keeping your own personal coding skills fresh and continuing to learn new ones.
- dofm 4mo ago
  I'm not that bothered about my coding skills, which are fine, and pretty up-to-date considering I'm now an old bloke. I am bothered about building an instinctive understanding that helps me deal with my anxieties and decide whether I want to carry on with this working life or quit. I needed to do this, this way, in my own time, to put my brain back together. It has worked for me, which is why I recommend it. YMMV.
  - ricardobayes 4mo ago
    Unfortunately the local llm bunch is not the most emphatetic one in my experience: you are somehow "expected" to immediately know all this stuff and god forbid you ask the wrong question. I've never seen or felt this level of bullying and weird vibes over tools and LLM models. "My setup works for you or beat it".
    - dofm 4mo ago
      There's also a lot of cargo-cult stuff, isn't there? Especially in the Reddit groups. Just do XYZ. And people ask why and they are never around to explain. Because, perhaps, they can't. (Very reminiscent of 3D printing, where you get a lot of very trivial advice poorly applied, which is an analogy I've now made several times.) Several of the youtubers are pretty helpful, though; I watched half a dozen things and absorbed the broad pattern and then went for it. Also I got a lot out of reading HN comments, which is why I am here; tucked away in the corners of these discussions are people who can help. Over time I hope I am one.
VerifiedReports 4mo ago
Exactly. The distinction between the various layers in "AI" systems is pretty vague to the newcomer. What is the "model" vs. the engine "running" it vs. weights? I don't recall any previous tech stack that was barfed onto the scene with so little background or reference material, going from zero to endless undefined jargon... and no primer in sight. For people who demand an understanding of their tools, it's a lot of work. I recognize the value of "AI" in performing the tasks I'd have to do manually; for example, keeping the data structures of my front- and back-ends in sync in a project. But do I want to interrupt my development and take weeks off to digest all of these tools? And if I do, I want to run the show and fully understand it. And like you, I think that's best done locally.
- ricardobayes 4mo ago
  For the most part you can just download LM Studio and go from there. It provides a chat interface and an easy-to-use interface to browse, load and use LLM models. The engine: it is abstracted away by LM Studio, if you want to dig deep it's llama.cpp as the runtime. Weights are the files what you download, they are the models for practical purposes.
  - dofm 4mo ago
    I definitely would recommend LM Studio as a learning environment, because it surfaces a bunch of things in relatively clear-minded ways. I am very grateful for it.
- Fr0styMatt88 4mo ago
  The most unexpected thing for me was kind of philosophical in a ‘holy shit’ way. Cloud models still feel ‘magic’, like you send a request off and get something back, like it’s something ‘special’. I used to joke that ChatGPT might be some kind of mechanical turk underneath. Watching a model run local on your own machine hits different — you realise that yes, it IS just a computer program. Which for me actually makes me appreciate the leap we’ve made MORE, not less. From an information-theoretic point of view, LLMs really are something special. The fact that they are just programs, that I’ve now experienced first-hand that they’re just programs, makes all those questions around consciousness and intelligence much more interesting.
bsder 4mo ago
> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus. Hello, my brother, just know that you have a fellow passenger in life at the same age who thinks the same thing. I agree that the local stuff is helping my understanding a LOT. However, my gut feel as someone who got to experience the TeleBomb after the DotBomb is that the obfuscation is INTENTIONAL--it's neither you nor your age. I remember asking people to explain to me what the OC-768 startup endgame was when roughly 10 OC-768 links could carry the world's traffic at the time--and everybody giving me blank looks. The AI Bubble has the EXACT same feel as the Telecom Bubble--just bigger. What I really wish is that I could find a VPS-type provider where I could toss things into their NVIDIA/AMD machines for an hour or two. Alas, all of the providers seem to want massive paperwork and huge minimum purchases. I can't wait for the bubble to pop so that we mere mortals can finally build with this stuff.
- hughw 4mo ago
  You looked into vendors like https://www.runpod.io/ https://www.runpod.io/?
ehnto 4mo ago
It's also great to have capability to run local models for more brute force tasks. Because you can change the system prompt, you can get local LLMs to do all kinds of high volume tasks without burning through tokens on a hosted model. Just one example, I needed a bunch of images tagged and organised, with a local vision capable model I could pretty easily set that up and leave it running overnight. I already had the GPU and memory for gaming, so it was at no cost for me to start running local models. But I feel the long term writing is on the wall, local models will only make more and more sense as they get better and more efficient.
bpye 4mo ago
> The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Seems like a GPU with 12GB+ VRAM is going to be a much more affordable way to achieve that? Even a B580 should get reasonable perf there.
- dofm 4mo ago
  No idea. I am a Mac guy, have been for a very long time. I buy them secondhand as a rule. I guess I would build a powerful home LLM server if I was convinced I really needed one for my purposes for some agentic application or other. At the moment I'd prefer to ride this out with a machine that is also an excellent Mac.
m3kw9 4mo ago
What’s the use of a 4gb Gemma other than to just play with it ?
musebox35 4mo ago
Thanks for posting this. This is the tinkerer mentality. It is not for everyone, but certain things can only be learned in that way. It is the best antidote to AI paranoia. There is much that does not transfer between frontier models and local ones. There is that. But you can not tinker as much as you can with the former.

by dofm 4mo ago

rusk 4mo ago

> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. I found LM studio to be a nice starting point. Frindlier and more featureful than Ollama and not as intimidating as llama.cpp (though you will want to use that eventually)

dofm 4mo ago
LM Studio is also nice because of the way the interface explains things; parameters have explanations and hints. It has been designed by people who really care about making it understandable. I tried Ollama but I've settled on Unsloth Studio generally; once things really settle down I'll just run the llama-server UI, which is pretty nice. A friend is tinkering with LLMs for amusement on a 16GB Raspberry Pi 5, and when I explained that llama.cpp now had a typical web chat interface he was so happy — it's amazing what the "table stakes" are now.

ddalex 4mo ago

I just got Claude to download and install all the models and servers and agents and prepare all the launch scripts for me... no need to learn, just ask it to do it for you

kdkdjduxnd 4mo ago
[dead]
dofm 4mo ago
Right, but I am a middle-aged bloke who is experiencing existential angst about whether I can carry on in this industry. I have a pretty deep, maybe paranoid need to be confident I have an intrinsic understanding, and I have found in my life that lessons come to you when you make yourself open to learning. So I need to build on top of what I know, taking as much of the hard way as I can bear to take at any one time — it has to be not quite difficult enough to put me off. I can't really explain what I have learned this way that is different, but I feel it in a way that I wouldn't if I'd simply pushed a button. For the same reason, I have a really basic 3D printer that I've set up myself, set up Klipper, configured how I want it, learned how to calibrate, all that. And now I can say that I feel I have an understanding of 3D printing. I could hold my head above water in a discussion with a real expert, maybe find work in an adjacent field where my insights would keep me grounded. I can afford a really good printer that has all that set up, and more, has no problems. But I'd just be someone who has a 3D printer. (Also who am I kidding about the existence of a printer with no problems)
- greyskull 4mo ago
  This really resonates with me, and I'm only a decade and change into my career. I use claude a lot day to day. I try to use it sensibly, making me more productive and produce better work. I'm also trying not to lose understanding along the way. I want to be able to actually talk to the conclusions I'm reaching. I have colleagues that seem perfectly content to delegate too much to the agents, and it saddens me. It feels like there will be swaths of engineers that didn't train some of the critical thinking skills that I take for granted. I certainly see it in slack discourse around anything more complicated than a feature implementation. Maybe I'm just cynical. Time will tell, I suppose.
  - sanderjd 4mo ago
    For me (about halfway between you and dofm in my career by your own statements in this thread), it's a dream at the moment. I can delegate all the tedious stuff that I've done "the hard way" a thousand times already and feel I have very little of value remaining to learn, so that I can spend more time on all the things that are actually new and thus much more interesting.

cyanydeez 4mo ago

I've setup to local paradigms for local coding: - opencode with it's webui - deer-flow with it's research/powered front end They both run websites so you don't have to baby sit them (eg, keep your mac open). I've build a pdf compressor over a few days by first having deer flow try and research the frameworks and pipeline. It stalls out because its not really a fluid programmer. Once it stalls out, I transferred it (manually for now) to opencode and it's refactoring it because it's just a collective bundle of sticks and it needs a lot of testing to tweak out the limited scop context. LLMs can't really hold large scopes (locally anyway, from what I've read from HN, it's possible with longer context). It'll complete in a few days with maybe 3-4 hours of full attention interaction, but it's running 3x that without my attention. Obviously, if I paid more attention it'd run quicker, but since it's local, it's not pumping out large volumes of code, it's mostly looping over tests and capabilities as observed. It's running Qwen3.6 35B MoE on a AMD 128GB strix halo. If I switched to the dense models, perhaps it'd be smarter, but the trade off seems to be much slower gen.

dofm 4mo ago
> - opencode with it's webui Have you tried Paseo? I have opencode in a VM, and the paseo daemon running in the VM, and then the Paseo Mac app. Really nice. (You can also use the Opencode GUI to frame a remote opencode web interface)
- c-hendricks 4mo ago
  You can also just add OpenCode web as a PWA, if that's what you mean by "frame". I'm gonna check out paseo, but am not looking forward to all the ram the agent needs + all the ram paseo needs
  - c-hendricks 4mo ago
    Have checked out Paseo, not sure what it offers over opencode web though. Definitely seems great if you're using other harnesses, but it seems like all it has over opencode web is split views and native apps. Neither of those really matter to me, plus you lose some opencode goodies. The preview urls are a neat idea, but our dev servers at work are mostly port independent and required to be on a certain subdomain for auth.

pizza234 4mo ago

> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem. Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are currently an expensive toy. When I try to program with a local LLM (I'm on a 32/128 GB system), I end up wasting time compared to a cloud LLM.

dofm 4mo ago
Again, I would not argue against any of this. And I can't say that I won't switch to openrouter (even just for the same models) at some point. But one of the things I have found about my own process learning is that some lessons only come to you when you make yourself available to them. And if that means doing things the difficult way, that is what you should do.
- wahnfrieden 4mo ago
  Difficult... and wastefully expensive
  - dofm 4mo ago
    I mean, it's a (secondhand) computer I bought for other tasks (processing very large photos, compiling large apps quickly). It's running all the time. It can also run LLMs when I want to. The rest of my life is ultra-frugal so I am relaxed about this.
    - _puk 4mo ago
      Don't bite. You're right. Having spent a good weekend learning how to perform latent-steering through playing with pytorch and a local Gemma4 model, there is no way I could have groked any of that in the the way I did without hands on time. This is on an M3 Max 36GB I've had for a couple of years. No further outlay needed.

codazoda 4mo ago

I agree with the learning aspect, but I have another motivation. I suspect that closed models might become too expensive to run for personal hobbyist use. I’ve been planning to buy a 64GB machine just to allow the limited local models this enables.

oceanplexian 4mo ago

Honestly your best bet is to buy a $20 Claude subscription, ask Claude to set it all up with Pi and llama.cpp and come back in 20 minutes after a cup of coffee. This is also a good idea because it will help set expectations of what a local model can do vs. a frontier model.

mullen 4mo ago
This is what I did after struggling to get llama.cpp working at a decent speed on my M1 Macbook. The secret is to very specific with your needs and targeted in what you are using llama.cpp for. Mine setup is just about strictly for qwen3-coder and now, I get a fairly decent speed out of it. I also installed Cursor to check Claude and it all worked out well.
- trey-jones 4mo ago
  I don't know why we're even talking about Qwen3.6 for writing code when qwen3-coder exists. My experience is there's no contest. I'm using 30b with 96k context on a dedicated server.
  - fouc 4mo ago
    For agentic workflows like tool use, editing codebases, multi-turn debugging?
- kristianp 4mo ago
  Are you talking about Qwen3 Coder 30b a3b Instruct from August 2025, which is a non-reasoning model? Or the more recent "Qwen3 Coder Next" from Feb this year with 80b params, 3b active? I found Qwen3 coder next to be quite good on openrouter [1], but couldn't run it locally. [1] https://openrouter.ai/qwen/qwen3-coder-next https://openrouter.ai/qwen/qwen3-coder-next

ricardobayes 4mo ago

I'd say give it some time for the dust to settle. This field badly needs standardized benchmarks even before the conversation around model goodness can start.

not_kurt_godel 4mo ago

> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Agree having a powerful machine is really worth it in general for professionals, but strong disagree that running local LLMs has anything to do with it. It's hard enough as it is getting a good ROI on your time/money prompting/wrangling with frontier models. IMO leaning on the comparatively limited capabilities of local LLMs is best avoided in favor of keeping your own personal coding skills fresh and continuing to learn new ones.

dofm 4mo ago
I'm not that bothered about my coding skills, which are fine, and pretty up-to-date considering I'm now an old bloke. I am bothered about building an instinctive understanding that helps me deal with my anxieties and decide whether I want to carry on with this working life or quit. I needed to do this, this way, in my own time, to put my brain back together. It has worked for me, which is why I recommend it. YMMV.
- ricardobayes 4mo ago
  Unfortunately the local llm bunch is not the most emphatetic one in my experience: you are somehow "expected" to immediately know all this stuff and god forbid you ask the wrong question. I've never seen or felt this level of bullying and weird vibes over tools and LLM models. "My setup works for you or beat it".
  - dofm 4mo ago
    There's also a lot of cargo-cult stuff, isn't there? Especially in the Reddit groups. Just do XYZ. And people ask why and they are never around to explain. Because, perhaps, they can't. (Very reminiscent of 3D printing, where you get a lot of very trivial advice poorly applied, which is an analogy I've now made several times.) Several of the youtubers are pretty helpful, though; I watched half a dozen things and absorbed the broad pattern and then went for it. Also I got a lot out of reading HN comments, which is why I am here; tucked away in the corners of these discussions are people who can help. Over time I hope I am one.

VerifiedReports 4mo ago

Exactly. The distinction between the various layers in "AI" systems is pretty vague to the newcomer. What is the "model" vs. the engine "running" it vs. weights? I don't recall any previous tech stack that was barfed onto the scene with so little background or reference material, going from zero to endless undefined jargon... and no primer in sight. For people who demand an understanding of their tools, it's a lot of work. I recognize the value of "AI" in performing the tasks I'd have to do manually; for example, keeping the data structures of my front- and back-ends in sync in a project. But do I want to interrupt my development and take weeks off to digest all of these tools? And if I do, I want to run the show and fully understand it. And like you, I think that's best done locally.

ricardobayes 4mo ago
For the most part you can just download LM Studio and go from there. It provides a chat interface and an easy-to-use interface to browse, load and use LLM models. The engine: it is abstracted away by LM Studio, if you want to dig deep it's llama.cpp as the runtime. Weights are the files what you download, they are the models for practical purposes.
- dofm 4mo ago
  I definitely would recommend LM Studio as a learning environment, because it surfaces a bunch of things in relatively clear-minded ways. I am very grateful for it.
Fr0styMatt88 4mo ago
The most unexpected thing for me was kind of philosophical in a ‘holy shit’ way. Cloud models still feel ‘magic’, like you send a request off and get something back, like it’s something ‘special’. I used to joke that ChatGPT might be some kind of mechanical turk underneath. Watching a model run local on your own machine hits different — you realise that yes, it IS just a computer program. Which for me actually makes me appreciate the leap we’ve made MORE, not less. From an information-theoretic point of view, LLMs really are something special. The fact that they are just programs, that I’ve now experienced first-hand that they’re just programs, makes all those questions around consciousness and intelligence much more interesting.

bsder 4mo ago

> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus. Hello, my brother, just know that you have a fellow passenger in life at the same age who thinks the same thing. I agree that the local stuff is helping my understanding a LOT. However, my gut feel as someone who got to experience the TeleBomb after the DotBomb is that the obfuscation is INTENTIONAL--it's neither you nor your age. I remember asking people to explain to me what the OC-768 startup endgame was when roughly 10 OC-768 links could carry the world's traffic at the time--and everybody giving me blank looks. The AI Bubble has the EXACT same feel as the Telecom Bubble--just bigger. What I really wish is that I could find a VPS-type provider where I could toss things into their NVIDIA/AMD machines for an hour or two. Alas, all of the providers seem to want massive paperwork and huge minimum purchases. I can't wait for the bubble to pop so that we mere mortals can finally build with this stuff.

hughw 4mo ago
You looked into vendors like https://www.runpod.io/ https://www.runpod.io/?

ehnto 4mo ago

It's also great to have capability to run local models for more brute force tasks. Because you can change the system prompt, you can get local LLMs to do all kinds of high volume tasks without burning through tokens on a hosted model. Just one example, I needed a bunch of images tagged and organised, with a local vision capable model I could pretty easily set that up and leave it running overnight. I already had the GPU and memory for gaming, so it was at no cost for me to start running local models. But I feel the long term writing is on the wall, local models will only make more and more sense as they get better and more efficient.

bpye 4mo ago

> The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Seems like a GPU with 12GB+ VRAM is going to be a much more affordable way to achieve that? Even a B580 should get reasonable perf there.

dofm 4mo ago
No idea. I am a Mac guy, have been for a very long time. I buy them secondhand as a rule. I guess I would build a powerful home LLM server if I was convinced I really needed one for my purposes for some agentic application or other. At the moment I'd prefer to ride this out with a machine that is also an excellent Mac.

m3kw9 4mo ago

What’s the use of a 4gb Gemma other than to just play with it ?

musebox35 4mo ago

Thanks for posting this. This is the tinkerer mentality. It is not for everyone, but certain things can only be learned in that way. It is the best antidote to AI paranoia. There is much that does not transfer between frontier models and local ones. There is that. But you can not tinker as much as you can with the former.