6 ms·

Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't wor

by bmitc 2mo ago

Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command. I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.

xienze 2mo ago
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
- Auracle 2mo ago
  Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”? You could override, obviously.
  - zargon 2mo ago
    Yes, llama.cpp does that.
hypfer 2mo ago
What makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.
- porphyra 2mo ago
  > llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
  - bmitc 2mo ago
    This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX. For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
  - solenoid0937 2mo ago
    Sure, but front ends like LMStudio exist for that crowd Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are. It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
  - manquer 2mo ago
    Why would they wish to handcraft this ? That is what agents are for ? They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments. Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
    - hypfer 2mo ago
      > Why would they wish to handcraft this ? Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future. Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
CamperBob2 2mo ago
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
- deleted 2mo ago
  [deleted]
- bilekas 2mo ago
  Promise I'm not being flippant or rude, but why not ask it to write something like a parameterized script to do the same thing for you ?
  - CamperBob2 2mo ago
    Laziness, primarily. Plus, no two models are exactly alike, and some have different instructions in their model card or on the HF page.
    - bilekas 2mo ago
      Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.
    - deleted 2mo ago
      [deleted]
    - deleted 2mo ago
      [deleted]
- acka 2mo ago
  While you probably have the audacity to check the commands and code that your favorite LLM generates to complete this task, i'm pretty much certain that at some point in time, due to this pervasive mindset of "just let an AI agent do it," we'll have a front page news story of someone who had their family photos on their NAS uploaded to some public space, their security cameras connected to a "live porn for lolz" webring, and perhaps even their crypto wallets and bank accounts pilfered, all because their favorite LLM got exploited by some prompt injection attack while trying to be helpful doing system administration tasks. I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.
naasking 2mo ago
You know free LLMs can help you understand that command line or design your own...
skrebbel 2mo ago
> LM Studio doesn't work behind proxies. Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet? I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
- bmitc 2mo ago
  > I feel like that's a rather niche situation you got there Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
  - skrebbel 2mo ago
    Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic). Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
  - vardump 2mo ago
    Every big company? YMMV, but I'd say about 20-40% do.
  - ThreatSystems 2mo ago
    If you're on Linux you can probably use proxychains.
  - Aurornis 2mo ago
    > Every big company in the world uses a network proxy. It's becoming more rare, now. A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though. The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
Aurornis 2mo ago
Start by copying the command line from the Unsloth guides. You don’t need to fine tune all of those parameters to get started. It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal. > Ollama is confusing and doesn't seem to support Qwen3? Typing “Ollama qwen3” into Google takes you right to this page: https://ollama.com/library/qwen3 https://ollama.com/library/qwen3 If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
- bmitc 2mo ago
  I had seen this: > Attention: To be updated for Qwen3 on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.html https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not. I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.
losthubble 2mo ago
just tell claude/codex "set this up on my system $huggingfacelink"
kccqzy 2mo ago
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
freehorse 2mo ago
If you do not want to tweak parameters you can just run llama-server -m model.gguf That's it
bilekas 2mo ago
I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived. There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me. Even this command above, it's not clear where op got the model from. So I'm with yah. For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
- MrDrMcCoy 2mo ago
  Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.
  - bilekas 2mo ago
    But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page. If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
    - MrDrMcCoy 2mo ago
      What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?
  - bilekas 2mo ago
    Plain question for you, where can I find the gguf model of this to direct download ?
parthsareen 2mo ago
Hi! From Ollama here - you can run: ollama run qwen3.8 (or if on mac qwen3.8:27b-mlx)

by bmitc 2mo ago

xienze 2mo ago

Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.

Auracle 2mo ago
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”? You could override, obviously.
- zargon 2mo ago
  Yes, llama.cpp does that.

hypfer 2mo ago

What makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.

porphyra 2mo ago
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
- bmitc 2mo ago
  This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX. For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
- solenoid0937 2mo ago
  Sure, but front ends like LMStudio exist for that crowd Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are. It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
- manquer 2mo ago
  Why would they wish to handcraft this ? That is what agents are for ? They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments. Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
  - hypfer 2mo ago
    > Why would they wish to handcraft this ? Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future. Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.

CamperBob2 2mo ago

I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.

deleted 2mo ago
[deleted]
bilekas 2mo ago
Promise I'm not being flippant or rude, but why not ask it to write something like a parameterized script to do the same thing for you ?
- CamperBob2 2mo ago
  Laziness, primarily. Plus, no two models are exactly alike, and some have different instructions in their model card or on the HF page.
  - bilekas 2mo ago
    Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.
  - deleted 2mo ago
    [deleted]
  - deleted 2mo ago
    [deleted]
acka 2mo ago
While you probably have the audacity to check the commands and code that your favorite LLM generates to complete this task, i'm pretty much certain that at some point in time, due to this pervasive mindset of "just let an AI agent do it," we'll have a front page news story of someone who had their family photos on their NAS uploaded to some public space, their security cameras connected to a "live porn for lolz" webring, and perhaps even their crypto wallets and bank accounts pilfered, all because their favorite LLM got exploited by some prompt injection attack while trying to be helpful doing system administration tasks. I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.

naasking 2mo ago

You know free LLMs can help you understand that command line or design your own...

skrebbel 2mo ago

> LM Studio doesn't work behind proxies. Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet? I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.

bmitc 2mo ago
> I feel like that's a rather niche situation you got there Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
- skrebbel 2mo ago
  Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic). Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
- vardump 2mo ago
  Every big company? YMMV, but I'd say about 20-40% do.
- ThreatSystems 2mo ago
  If you're on Linux you can probably use proxychains.
- Aurornis 2mo ago
  > Every big company in the world uses a network proxy. It's becoming more rare, now. A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though. The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.

Aurornis 2mo ago

Start by copying the command line from the Unsloth guides. You don’t need to fine tune all of those parameters to get started. It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal. > Ollama is confusing and doesn't seem to support Qwen3? Typing “Ollama qwen3” into Google takes you right to this page: https://ollama.com/library/qwen3 https://ollama.com/library/qwen3 If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.

bmitc 2mo ago
I had seen this: > Attention: To be updated for Qwen3 on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.html https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not. I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.

losthubble 2mo ago

just tell claude/codex "set this up on my system $huggingfacelink"

kccqzy 2mo ago

There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.

freehorse 2mo ago

If you do not want to tweak parameters you can just run llama-server -m model.gguf That's it

bilekas 2mo ago

I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived. There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me. Even this command above, it's not clear where op got the model from. So I'm with yah. For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..

MrDrMcCoy 2mo ago
Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.
- bilekas 2mo ago
  But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page. If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
  - MrDrMcCoy 2mo ago
    What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?
- bilekas 2mo ago
  Plain question for you, where can I find the gguf model of this to direct download ?

parthsareen 2mo ago

Hi! From Ollama here - you can run: ollama run qwen3.8 (or if on mac qwen3.8:27b-mlx)