6 ms·
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure o
by dghlsakjg 3mo ago
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
- arjie 3mo agoHaha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
- hedora 3mo agoAMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.
- arjie 3mo agoProblem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
- apimade 3mo ago8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS. That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
- cududa 3mo agoJust a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
- xbmcuser 3mo agoWe will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
- formerly_proven 3mo agoExcept the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.
- zmmmmm 3mo agoSadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.
- jack_pp 3mo agoIsn't there such a thing as low hanging fruit? Aren't we already approaching theoretical physical limits? We're at 2nm
- kaashif 3mo ago(1) Yes. (2) Are you saying that you think we're at the limits of computing in general, or that specific technology? We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily. But that is...not a low hanging fruit.
- fragmede 3mo ago> We are nowhere near that. Nowhere?
- sh3rl0ck 3mo agoWe'd bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS. Not too wild an idea!
- gpt5 3mo agoMost people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case. Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
- anon373839 3mo agoI suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
- gpt5 3mo agoiPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people. However, the compute situation just skew it even further when it comes to LLMs.
- anon373839 3mo agoPhones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
- marci 3mo agoSeems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
- josu 3mo agoI don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
- l33tman 3mo agoCould it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?
- marci 3mo agoNothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks. https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models https://machinelearning.apple.com/research/introducing-third...
- josu 3mo agoThanks for responding.
- Perz1val 3mo ago> wearing the hard drive Do reads wear out nvme? Iirc only writes do
- efficax 3mo agoonly writes and honestly that’s a bit overblown they last a long time
- croes 3mo agoSo every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore. That time won’t be as much fun as you think
- cromka 3mo agoBy that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited. We'll be fine.
- bestham 3mo agoNo, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
- cromka 3mo agoThat's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
- JKCalhoun 3mo agoIf that's the natural outcome then the future you're describing is inevitable. Oh well, maybe the last one to leave the internet can turn the lights off.
- croes 3mo agoIt’s not a natural outcome it’s a decision
- myrmidon 3mo agoAgree with this. As soon as things get in range for motivated amateurs, progress skyrockets. Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware. I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see). I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
- Bayart 3mo ago> As soon as things get in range for motivated amateurs, progress skyrockets. That's how the AI boom came around to start with, before CUDA it was an academic topic.
- mannanj 3mo agoIs this where with in conjunction with Jevon's paradox, we are AI usage to exponentially skyrocket?
- RachelF 3mo ago>As soon as things get in range for motivated amateurs, progress skyrockets. So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption. This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.
- pizza234 3mo ago> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illusion that that will be possible.
- AlecSchueler 3mo ago> This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. That is, an essential step?
- rhdunn 3mo agoThere are several aspects to this: 1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.). 2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models. 3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s. 4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version). 5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert). 5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
- sipjca 3mo agoUnfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen. While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
- hagen8 3mo agoThere are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
- jarek83 3mo agoIf 1t param model would run from SSDs those will stop costing only $200.
- JKCalhoun 3mo agoAgree—and I want to see this functionality move to the larger platforms that everyone uses like LM Studio.
- ac29 3mo agollama.cpp has used mmap by default for years
- ksec 3mo ago>when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
- spacebacon 3mo ago[dead]