4 ms·

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Cl

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

stymaar 1mo ago
> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
- Schiendelman 1mo ago
  Your mileage may vary. I tried this a couple months ago and spent a full day on it just not working before giving up. Anything I sent, it wouldn't run.
- fxtentacle 1mo ago
  My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.
  - WhyNotHugo 1mo ago
    Not necessarily rebuild llama from scratch, build attempting to build it without cmake and manually invoking all the build commands would be quite in character.
  - catlifeonmars 1mo ago
    I wonder if this is an artifact of RL, where the training heavily emphasizes codegen. It may be that the model is just better at generating code than reusing libraries, so it prefers the lowest cost approach. I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api
- raffraffraff 1mo ago
  I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
  - funlang 1mo ago
    [flagged]
    - microtonal 1mo ago
      Add Just [...] And you have also fully adopted the writing patterns.
      - lelanthran 1mo ago
        Seems like he's self-aware, though - he's taking steps to move away from brain-atrophy.
        bretpiatt 1mo ago
        My ability to detect sarcasm is not good. From looking at funlang's profile and other comments the profile looks like a LLM generated bot. Forums with full no verification pseudonyms seem like they have a real challenge ahead. How long until we need humanhackernews.com with public pseudonyms and a private trusted verification?
        funlang 1mo ago
        [flagged]
  - enraged_camel 1mo ago
    To me laziness has nothing to do with it. I'd rather focus on my limited time on things that actually matter, e.g. the design of an important product feature or the root cause of a bug and the proper fix. I delegate everything else to Claude. It's really nice to be able to say to Fable "see if you can get this running locally" and then come back 10-30 mins later and read the results. I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
- LeBit 1mo ago
  In the README there is usually a section where it explains which llama.cpp parameters to configure (sampling, penalties, context size, etc) to get better results.
raffraffraff 1mo ago
Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason). If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
- petu 1mo ago
  There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
  - selcuka 1mo ago
    > There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Probably this: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill https://huggingface.co/empero-ai/Qwen3.8-4B-Distill > Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
    - formerly_proven 1mo ago
      That's not distillation, that's model homeopathy.
wccrawford 1mo ago
That's funny, because I just went through the opposite. I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions. I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it. Then I got fed up and asked Claude. Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed. Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working. Then it tuned it somewhat. Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working. In 30 minutes. It was absolutely amazing. Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching. But Claude got it done for me. Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?
- ohyes 1mo ago
  Am I the only one who just downloads directly from LM Studio and just runs the server there? It’s trivial.
  - reverius42 1mo ago
    You're not the only one, I'm running Qwen3.8-27B in LM Studio and it seems to be going great. Was very easy to set up.

stymaar 1mo ago

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

Schiendelman 1mo ago
Your mileage may vary. I tried this a couple months ago and spent a full day on it just not working before giving up. Anything I sent, it wouldn't run.
fxtentacle 1mo ago
My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.
- WhyNotHugo 1mo ago
  Not necessarily rebuild llama from scratch, build attempting to build it without cmake and manually invoking all the build commands would be quite in character.
- catlifeonmars 1mo ago
  I wonder if this is an artifact of RL, where the training heavily emphasizes codegen. It may be that the model is just better at generating code than reusing libraries, so it prefers the lowest cost approach. I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api
raffraffraff 1mo ago
I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
- funlang 1mo ago
  [flagged]
  - microtonal 1mo ago
    Add Just [...] And you have also fully adopted the writing patterns.
    - lelanthran 1mo ago
      Seems like he's self-aware, though - he's taking steps to move away from brain-atrophy.
      - bretpiatt 1mo ago
        My ability to detect sarcasm is not good. From looking at funlang's profile and other comments the profile looks like a LLM generated bot. Forums with full no verification pseudonyms seem like they have a real challenge ahead. How long until we need humanhackernews.com with public pseudonyms and a private trusted verification?
        funlang 1mo ago
        [flagged]
- enraged_camel 1mo ago
  To me laziness has nothing to do with it. I'd rather focus on my limited time on things that actually matter, e.g. the design of an important product feature or the root cause of a bug and the proper fix. I delegate everything else to Claude. It's really nice to be able to say to Fable "see if you can get this running locally" and then come back 10-30 mins later and read the results. I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
LeBit 1mo ago
In the README there is usually a section where it explains which llama.cpp parameters to configure (sampling, penalties, context size, etc) to get better results.

raffraffraff 1mo ago

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason). If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.

petu 1mo ago
There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
- selcuka 1mo ago
  > There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Probably this: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill https://huggingface.co/empero-ai/Qwen3.8-4B-Distill > Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
  - formerly_proven 1mo ago
    That's not distillation, that's model homeopathy.

wccrawford 1mo ago

That's funny, because I just went through the opposite. I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions. I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it. Then I got fed up and asked Claude. Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed. Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working. Then it tuned it somewhat. Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working. In 30 minutes. It was absolutely amazing. Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching. But Claude got it done for me. Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?

ohyes 1mo ago
Am I the only one who just downloads directly from LM Studio and just runs the server there? It’s trivial.
- reverius42 1mo ago
  You're not the only one, I'm running Qwen3.8-27B in LM Studio and it seems to be going great. Was very easy to set up.