9 ms·
This one is pretty impressive. I'm running it on my Mac via Ollama - only a 20GB download, tokens spit out pretty fast and my initial prompts have shown some go
by simonw 2y ago
This one is pretty impressive. I'm running it on my Mac via Ollama - only a 20GB download, tokens spit out pretty fast and my initial prompts have shown some good results. Notes here: https://simonwillison.net/2024/Nov/27/qwq/ https://simonwillison.net/2024/Nov/27/qwq/
- cherioo 2y agoWhat hardware are you able to run this on?
- simonw 2y agoM2 MacBook Pro with 64GB of RAM.
- naming_the_user 2y agoWorks well for me on an MBP with 36GB ram with no swapping (just). I've been asking it to perform relatively complex integrals and it either manages them (with step by step instructions) or is very close with small errors that can be rectified by following the steps manually.
- torginus 2y agoSorry for the random question, I wonder if you know, what's the status of running LLMs non-NVIDIA GPUs nowadays? Are they viable?
- danielbln 2y agoApple silicon is pretty damn viable.
- throwaway314155 2y agoPretty sure they meant AMD
- torginus 2y agoYeah, but if you buy ones with enough RAM, you're not really saving money compared to NVIDIA, and you're likely behind in perf.
- anon373839 2y agoNvidia won’t sell these quantities of RAM at Apple’s pricing. An A100 80GB is $14k, while an M3 Max MBP with 96GB of RAM can be had for $2.7k.
- paxys 2y ago96GB of unified RAM. How much of that is available to the graphics cores? I haven't tested a later model but the M1 Max would max out at 16GB VRAM regardless of how much the machine had. There's a reason companies are setting up clusters of A100s, not MacBooks.
- fragmede 2y agohttps://techobsessed.net/2023/12/increasing-ram-available-to-gpu-on-apple-silicon-macs-for-running-large-language-models/ https://techobsessed.net/2023/12/increasing-ram-available-to... says it's tunable via the terminal down to 2GiB for reserved for the OS with the rest allocated for GPU use.
- fennecfoxy 2y agoNot only that but Apple's ram is 0.5TB/s pretty much, a 4090 gets 1TB/s. I feel like the discrete card is the better value proposition because: nobody should need to be running 80GB models on a laptop, I feel this is more in the high perf/research area, you could argue that it could be a useful tool as a co-pilot but you've tuned your machine to use all ram for the model...you can't do anything else. Additionally, it's such a specific use case for the machine that trying to sell it would be hard, whereas I can hock off a GPU to someone doing data, ML, gaming, video editing, etc.
- mldbk 2y agoI run llama on 7900XT 20GB, works just fine.
- mark_l_watson 2y agoI am running it on a 32G memory mac mini with an M2 Pro using Ollama. It runs fine, faster than I expected. The way it explains plans for solving problems, then proceeding step by step is impressive.
- j0hnyl 2y agoHow many tokens per second?
- mark_l_watson 2y agoI am away from my computer, but I think it was about 10/second - not too bad.
- andrewmunsell 2y agoAnother data point: 17.6 tokens/s on an M4 Max 40 core GPU
- behnamoh 2y ago8.4 tps on M1 Pro chip with 32GM RAM (Q4 model, 18GB).
- Terretta 2y agoIf your job or hobby in any way likes LLMs, and you like to "Work Anywhere", it's hard not to justify the MBP Max (e.g. M3 Max, now M4 Max) with 128GB. You can run more than you'd think, faster than you'd think. See also Hugging Face's MLX community: https://huggingface.co/mlx-community https://huggingface.co/mlx-community QwQ 32B is featured: https://huggingface.co/collections/mlx-community/qwq-32b-preview-67485e0db78b18207ba61359 https://huggingface.co/collections/mlx-community/qwq-32b-pre... If you want a traditional GUI, LM Studio beta 0.3.x is iterating on MLX: https://lmstudio.ai/beta-releases https://lmstudio.ai/beta-releases
- msk-lywenn 2y ago4699$US. Quickest justification I ever made not to buy something
- jstummbillig 2y agoTo add to that, given the wild trajectory of the field, it's at the very least doubtful, that that's going to buy you access to hardware (let alone: a model) that's still even remotely desirable for relevant AI use, even a year from now.
- tourmalinetaco 2y agoBut it has the logo I’ve been told to buy, so regardless of the quality or specs I need to buy it, how else will I ever show my face in Zoom meetings to my SoCal colleagues?
- fennecfoxy 2y agoYeah, exactly. If you don't want a discrete GPU of your own in a desktop that you could technically access remotely, then go with cloud GPU. Why pay Apple silly money for their ram when you could take that same money, get a MB Air and build a desktop with a 4090 in it (hell, if you already have the desktop you could buy TWO 4090s for that money). Then just set up the server to use remotely.
- deleted 2y ago[deleted]
- singularity2001 2y agouhm the pelican SVG is ... not impressive
- tethys 2y agoFor comparison, this is what other models produce: https://github.com/simonw/pelican-bicycle/blob/main/README.md https://github.com/simonw/pelican-bicycle/blob/main/README.m...
- mhast 2y agoThese are language models, they are not designed for producing image output at all. In a way it's impressive it can even produce working SVG code as output. Even more sonthst it vaguely resembles a bird on a bike.
- kfajdsl 2y agoYou try writing an SVG by hand without being able to look at the render.
- m3kw9 2y agoThe svg is very unimpressive but you are impressed by it, what gives? It looks nothing like a pelican
- simonw 2y agoAsking language models to draw things by outputting SVG is a deliberately absurd task. Given how unreasonable that is I thought this model did very well, especially compared to others that I've tried: https://github.com/simonw/pelican-bicycle?tab=readme-ov-file#pelicans-on-a-bicycle https://github.com/simonw/pelican-bicycle?tab=readme-ov-file...
- sams99 2y agoI find it odd that is refused me so badly https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai-conversations/xvsFWmLNYnT5B8wikJciWQ https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai... my guess is that I am using a quantized model It simply did not want to use XML tools for some reason something that even qwen coder does not struggle with: https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai-conversations/njzVS92kTuB4smkYZsdNBQ https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai... I have not seen any model including sonnet that is able to 1 shot a working 9x9 go board For ref gpt-4o which is still quite bad https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai-conversations/CpbtV7tHZ_boiHSwwKaevQ https://discuss.samsaffron.com/discourse-ai/ai-bot/shared-ai...