5 ms·
Looking forward to more open weight releases from Qwen, especially 122B and 397B.
by tarruda 5mo ago
Looking forward to more open weight releases from Qwen, especially 122B and 397B.
- smcleod 5mo agoYeah that 60-150b~ range is such a sweet spot for current 'prosumer' hardware, I'd love to see something like a 120b-a14b or there about.
- gcr 5mo agoWhat’s the price point for getting into that sweet spot? I’m on an M1 Max with 32GB VRAM, so I’m looking forward to the 27B or 35B-A3B models. Is dropping $5k for an RTX 6000 or a DGX Spark really the best option?
- tarruda 5mo ago> What’s the price point for getting into that sweet spot? In October/2024 I got my Mac studio M1 ultra with 128G, IIRC it was ~$2500. With recent prices explosion, it has certainly gotten more expensive. https://frame.work/ https://frame.work/ is selling 128G strix halo mainboard for $2700, but you have to add storage and case.
- ttoinou 5mo agoM5 Max 64GB (sweet spot) or 128GB (only 1000 USD, better to keep it for the future) more are the best quality price ratio, future proof, reliable, resellable and flexible workloads. Harder to use as a server might be the only drawback
- roger_ 5mo agoM5 Max 128GB for $1k?
- smallerize 5mo agoI think they mean the upgrade to 128GB is +$1k.
- tempoponet 5mo agoThe memory upgrade is $1k on a Macbook Pro. The laptop is ~$5500.
- throwaw12 5mo agoWhat do you recommend for non-Mac setup? I am a Mac user, but its getting expensive, and not seeing reason to jump to the latest M5
- varispeed 5mo agoProbably a comparable non-Mac setup will be Threadripper, but it will become much more expensive. My view is that actually Apple products are the cheapest on the market when it comes to performance.
- barbacoa 5mo agoTry looking into Ryzen AI Max 395. AMD made a CPU/GPU soc with unified memory specifically for ai inference. Can buy mini PCs with up to 128gb ram.
- simple10 5mo agoThe Ryzen AI Max 395 128gb is super cool, but not fast for inference. Order of magnitude slower than dedicated GPU but at half the cost. You can run larger models on it but it's slow. Great for local async work. Not great for daily chat or code agent driver.
- throwa356262 5mo agoThe latest NPUs are pretty fast, I think what is missing is more optimised software support.
- plagiarist 5mo agoThe vRAM bandwidth is at least as much a problem as compute on these ones, there is a lot of data to shuffle around
- krzyk 5mo agoIsn't CUDA/nvidia the go to solution for most local models, with the rest being second class citizents?
- anonym29 5mo agoStrix Halo at $2k with similar TG and about half the PP of DGX Spark was a pretty good deal IMO, especially considering it's also a full x86 system... 16c/32t Zen 5, 40 CU RDNA 3.5, 128 GB unified memory at ~220 GB/s real-world speeds (256 GB/s theoretical) - that runs full tilt at 140W in performance mode and idles at ~10W. Unfortunately, the prices rose on these a lot, but unevenly. Beelink GTR 9 Pro is $4400, Framework Desktop is ~$3500, for what is basically the exact same mainboard as a Bosgame M5 for $2800. Apple's M5 Max is another attractive option. Apple silicon traditionally had great MBW and was good at TG, but struggled with PP, but the new neural engines in those GPU cores have made a big difference in a good way here. Gorgon Halo is rumored for June announcement with Q4'26 release with basically +100 MHz clocks on Strix Halo, LPDDR5X-8533 instead of LPDDR5X-8000, but more importantly, 192 GB max instead of 128 GB. I'd say it's better to wait for Gorgon Halo than to grab Strix Halo now. However, Medusa Halo, rumored for H2'27, is slated to have up to 26c Zen 6 (heterogeneous cores - kinds funny that AMD is heading towards these as Intel retreats from them), 48 CU of RDNA 5 instead of 40 CU RDNA 3.5, and a 384 bit bus w/ LPDDR6, which should make 256 GB at more like ~490-600 GB/s MBW, which will really make Strix and Gorgon Halo obsolete. Also worth keeping an eye out for Serpent Lake (intel CPU + nvidia iGPU on a single board with unified memory, rumored for 2028-2029 iirc), and on the 160 GB Crescent Island Intel dGPU.
- tempoponet 5mo agoExpect to pay $4k-10k - Your RTX 6000 is closer to $10k now - Sparks are creeping into the $4-5k range - AMD Strix are ~3.5k - Apple depends on chipset and memory. Sweet spot would be 128gb M3 Ultra, probably $6-8k but admittedly haven't been tracking closely. New M5 might come in the fall. You can get a new 128gb M5 Max laptop for ~5-6k today. - a 4x3090 rig would take $5-6k Every platform has tradeoffs, but it's mostly ecosystem, memory bandwidth, and power consumption. They're all slow. The best option is likely to rent hardware on Runpod. The RIO on self-hosting is very low unless you have a specific need or you're ok treating it as a hobby.
- anonym29 5mo agoBosgame M5 (Strix Halo) w/ 128 GB still goes for $2800 right now. SH systems have surged in price dramatically but quite unevenly. >The best option is likely to rent hardware on Runpod. Vast.ai is much cheaper, but the broader point here is contestable. The only dimension in which cloud GPU rentals win is cost. You lose the confidentiality, integrity, and availability benefits of local deployments.
- ai_fry_ur_brain 5mo agoRentals are priced to pay themselves off in 1-1.5 years (when renting them out per hour, not selling tokens). Its never a better option to rent. Not that I'd encourage anyone to throw large amounts of money to have access to LLMs, but you're definately going to be better off buying something that you can amortize over multiple years with a multi year warranty.
- ai_fry_ur_brain 5mo agoAnd for what? Spend 10-15k for the slopiest of slop code, non deterministic automations, and the ability to spawn an AI gf? This whole thing is really starting to remind me of the crypto hype phases of 2016-2018 when everyone thought their investment in GPUs was going to make them rich.
- organsnyder 5mo ago
- embedding-shape 5mo agoIf I could find a RTX Pro 6000 for $5K I'd definitively grab it, I'm running RedHatAI/Qwen3.6-35B-A3B-NVFP4 on one (I had to pay closer to $10K for it though) with 260K context and it's a blast! ds4 by antirez also works well, even IQ2XXS seems to work relatively well but Qwen3.6-35B-A3B-NVFP4 is both faster and higher quality responses (at least for coding and translations which I use them mostly for).
- pulse-dev 5mo ago[dead]
- smcleod 5mo agoReally right now it's the M5 Max MacBook Pro 128GB, the RTX6000 is a nice card but you'd need more than one of them and you have to have a desktop to suit. The DGX Spark is slow and has pretty limited software support.
- tandr 5mo agoDon't mind me asking, but where did you find $5k RTX 6000? Even 48GB model (previous gen) shows minimum at 7k, and 96GB one (Blackwell) is ~10k on Amazon...
- CamperBob2 5mo ago$5K is presumably what it costs to pay some local gangsters to break into an nVidia warehouse. That's the only you will pay $5K for an RTX 6000 for the next couple of years. The server edition has gone up $2K in the last couple of weeks alone, at the outlet where I bought one previously.
- tandr 5mo agoMan... Does it mean that buying RTX 6000 for 10k today is actually becoming an investment?
- tarruda 5mo agoI have a 128G mac studio and even 397B was a happy surprise to me due to its high quantization resilience. I've created a 2.54BPW quant that fit on my hardware with 128k context, 20 tps tg and 200tps pp, while maintaining high scores on many benchmarks: https://huggingface.co/tarruda/Qwen3.5-397B-A17B-GGUF/discussions/1#69d142b4f17676f98e53c16a https://huggingface.co/tarruda/Qwen3.5-397B-A17B-GGUF/discus...
- ttoinou 5mo agobetter than antirez ds4 ?
- tarruda 5mo agoI only tried a very early version of that when it was just a llama.cpp fork and Qwen was certainly better in my tests. But I was not super impressed with deepseek 4 flash using it from the official API either, so it doesn't seem quantization fault. It is a good model, but nothing out of the ordinary in the few benchmarks I ran on it (with full awareness that benchmarks are biased).
- chrisweekly 5mo agoApple store's current options for mac studio seem to max out at 96GB. I'm questioning ROI, esp. given it's not upgradeable. Curious about others' takes on new mac hardware.
- tarruda 5mo ago> I'm questioning ROI If by ROI you mean saving more money than using paid APIs, then I don't think it is worth it. All you gain is full sovereignty over your AI usage.
- drob518 5mo agoCurrently, Apple is letting some of its models go out of stock in preparation for new models coming in a few weeks. I would expect at least 128 GB models at that time. That said, the memory crunch is hitting everyone.
- KronisLV 5mo agoThere definitely have been some options in the past, cool to see them. Oddly enough, though, Qwen 3.6 35B A3B and Gemma got some really good reviews, despite being way smaller than any of these ones. Qwen 3.5, 122B A10B: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF Qwen Coder Next, 80B A3B: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF It's kinda weird that DeepSeek V4 Flash is supposed to be 284B A13B, but shows up as 158B in HuggingFace, probably some weird bug: https://huggingface.co/unsloth/DeepSeek-V4-Flash https://huggingface.co/unsloth/DeepSeek-V4-Flash and that's not even just Unsloth but like the official source too https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash (so also doesn't fit the category unless you get a heavily quantized version to run, but cool regardless) Mistral Medium 3.5 is interesting because it's 128B but dense, so probably too slow for most folks: https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF GPT-OSS, 120B A5B: https://huggingface.co/unsloth/gpt-oss-120b-GGUF https://huggingface.co/unsloth/gpt-oss-120b-GGUF
- mixtureoftakes 5mo agoI'm more excited for qwen3.7 9b and 72b, these are usually so good for their size
- guitcastro 5mo agoI am still waiting for qwem image-edit 2.0 open weight
- Pxtl 5mo agoOuch. I'm just getting into tinkering with these things - mine is running on a vanilla gaming desktop with a 12gb 3060 and 32gb of ram. Even going above Qwen 9B risks completely locking up the machine.
- ricardobayes 5mo agoPersonally even more a lower quantized model like 9B.
- throwa356262 5mo agoSame here, the unsloth versions can run on a potato and are actually useful.