3 ms·
It really is cursed to be spending hundreds of watts of power in a datacenter somewhere to make a laptop run slightly faster.
by 2001zhaozhao 10mo ago
It really is cursed to be spending hundreds of watts of power in a datacenter somewhere to make a laptop run slightly faster.
- ImPrajyoth 10mo agooh absolutely. burning a coal plant to decide if i should close discord is peak 2025 energy. strictly speaking, using the local model (Ollama) is 'free' in terms of watts since my laptop is on anyway, but yeah, if the inefficiency is the art, I'm the artist.
- bdhcuidbebe 10mo ago> using the local model (Ollama) is 'free' in terms of watts since my laptop is on anyway Now that’s a cursed take on power efficency
- ImPrajyoth 10mo agoefficiency is just a mindset. if i save 3 seconds of my own attention by burning 300 watts of gpu, the math works out in my favor!
- abeyer 10mo ago"works out in my favor" is a pretty poor metric. If I burn a billion tons of someone else's coal to make myself a paperclip (and don't have to breathe the outputs) it works out in my favor too.
- fragmede 10mo agoRunning ollama to compute inference uses energy that wouldn't have been used if you weren't running ollama. There's no free lunch here.
- redfloatplane 10mo agoAn interesting thought experiment - a fully local, off-grid, off-network LLM device. Solar or wind or what have you. I suppose the Mac Studio route is a good option here, I think Apple make the most energy efficient high-memory options. Back of the napkin indicates it’s possible, just a high up front cost. Interesting to imagine a somewhat catastrophe-resilient LLM device…
- ImPrajyoth 10mo agoThat is the endgame. I think we are moving toward a bilayered compute model: The Cloud: For massive reasoning. The Local Edge: A small, resilient model that lives on-device and handles the OS loop, privacy, and immediate context. BrainKernel is my attempt to prototype that Local Edge layer. Its messy right now, but I think the OS of 2030 will definitely have a local LLM baked into the kernel.
- redfloatplane 10mo agoWell, on my Macbook, some of that already exists. In the Shortcuts app you can use the "Use Model" action which offers to run an LLM on apple's cloud, on-device, or other external service (eg ChatGPT). I use this myself already for several actions, like reading emails from my tennis club to put events in my calendar automatically. Whether or not we'll see it lower down in the system I'm not sure. Honestly I'm not certain of the utility of an autonomous LLM loop in many or most parts of an OS, where (in general) systems have more value the more deterministic they are, but in the user space, who can say. In any case, I certainly went down a fun rabbit hole thinking about a mesh network of LLM nodes and thin clients in a post-collapse world. In that scenario, I wonder if the utility of LLMs is really worth the complexity versus a kindle-like device with a copy of wikipedia...
- evilduck 10mo agoMacs would be the most power efficient with faster memory but an AI Max 395+ based system would probably be the most cost efficient right now. A Framework Desktop with 128GB of shared RAM only pulls 400W (and could be underclocked) and is cheaper by enough that you could buy it plus 400W of solar panels and a decently large battery for less than a Mac Studio with 128GB of RAM. Unfortunately the power efficiency win is more expensive than just buying more power generation and storage ability.
- nubinetwork 10mo agoAn entire datacenter on the other hand, might be appealing to spot things you wouldn't otherwise see in a sea of logs and graphs.