4 ms·

How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't opti

by xrd 4mo ago

How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730 https://github.com/ggml-org/llama.cpp/issues/24730

jeremyjh 4mo ago
No one is doing that for a model this size it would have to be so heavily quantized that it wouldn’t be useful - or you’d need to spend a half million dollars on hardware. People use hosted APIs. Open weight means cloud vendors can host it.
- malshe 4mo ago
  Can you recommend any US based cloud providers?
  - maybe_pablo 4mo ago
    In HuggingChat (https://huggingface.co/chat https://huggingface.co/chat) you can test open models for free and even test specific providers. From there I collected the following US providers currently serving GLM 5.2: - Together (https://www.together.ai/models https://www.together.ai/models) - Fireworks (https://fireworks.ai/models https://fireworks.ai/models) - Featherless (https://featherless.ai/models https://featherless.ai/models)
    - malshe 4mo ago
      That's great. Thank you!
  - fooster 4mo ago
    ollama cloud, neuralwatt.
    - malshe 4mo ago
      Thanks