3 ms·

128GB RAM Ryzen AI MAX+, $1699 – Bosman Undercuts All Other Local LLM Mini-PCs

43 points by mdp2021 1y ago

billconan 1y ago
is its RAM upgradable?
- magicalhippo 1y ago
  I would be very surprised. Typically LPDDR is soldered, as it takes too much power to run the traditional sockets, as well as being much slower. Though there has been a modular option called LPCAMM[1]. However AFAIK it doesn't support the speed the specs of this box states. Recently a newer connector, SOCAMM has been launched[2], which does support the high memory speeds, but it's just on the market and going into servers first AFAIK. [1]: https://www.anandtech.com/show/21069/modular-lpddr-becomes-a-reality-samsung-introduces-lpcamm-memory-modules https://www.anandtech.com/show/21069/modular-lpddr-becomes-a... [2]: https://www.tomshardware.com/pc-components/ram/micron-and-sk-hynix-unveil-lpddr5x-socamm-up-to-128gb-for-ai-servers https://www.tomshardware.com/pc-components/ram/micron-and-sk...
  - duskwuff 1y ago
    SOCAMM is also Nvidia-specific, not a wider standard. (At least, not yet.)
  - aitchnyu 1y ago
    Will this save upgradable RAM on laptops? At the same time, dual channel is needed and laptops give only 1 slot to upgrade.
    - magicalhippo 1y ago
      Good question. Perhaps for higher-end models. Though cost, weight and physical space still weighs in favor of soldered RAM.
- hnuser123456 1y ago
  No, it's soldered, it would have to run around 6000 MT/s instead of 8533 if it was slotted DIMMs.
3eb7988a1663 1y ago
If you are doing nothing but consuming models via llama.cpp, is the AMD chip an obstacle? Or is that more a problem for research/training where every CUDA feature needs to be present?
- _lvbh 1y ago
  Llama.cpp works well on AMD, even for really outdated GPUs. Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan.
  - Havoc 1y ago
    >Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan. That's a bit odd given ollama utilizing llama to do the inference...
    - LorenDB 1y ago
      See recent discussion about this very topic: https://news.ycombinator.com/item?id=42886680 https://news.ycombinator.com/item?id=42886680
    - DiabloD3 1y ago
      It isn't odd at all. Ollama uses an ancient version of llama.cpp, and was originally meant to just be a GUI frontend. They forked, and then never resynchronized... and now lack the willpower and technical skill to achieve that. Ollama is essentially a dead, yet semi-popular, project with a really good PR team. If you really want to do it right, you use llama.cpp.
  - washadjeffmad 1y ago
    Don't you dare say anything unpositive about Ollama this close to whatever it is they're planning to distinguish themselves from llama.cpp. They've been out hustling, handshaking, dealmaking, and big businessing their butts off, whether or not they clearly indicate the shoulders of the titans like Georgi Gerganov they're wrapping, and you are NO ONE to stand in their way. Do NOT blow this for them. Understand? They've scooted under the radar successfully this far, and they will absolutely lose their shit if one more peon shrugs at how little they contribute upstream for what they've taken that could have gone to supporting their originator. Ollama supports its own implementation of ggml, btw. gglm is a mysterious format that no one knows the origins of, which is all the more reason to support Ollama, imo.
ilaksh 1y ago
How does this sort of thing perform with 70b models?
- hnuser123456 1y ago
  273 GB/s / 70GB = 3.9 tokens/sec
  - mdp2021 1y ago
    Are you sure that kind of computation can be a general rule? Did you mean that the maximum rate it could be obtained is "bandwidth/size"?
    - deleted 1y ago
      [deleted]
    - hnuser123456 1y ago
      Yes, for most LLMs the transformer means the entire model and context is read from VRAM for every token.
olddustytrail 1y ago
That's an odd coincidence. I'd decided to get a new machine but I suspected we'd start seeing new releases with tons of GPU accessible RAM as people want to experiment with LLMs. So I just got a cheap (~350 USD) mini PC to keep me going until the better stuff came out. Which was a 24GB, 6c/12t CPU from a company I'd not heard of called Bosgame (dunno why the article keeps calling them Bosman unless they have a different name in other countries. It's definitely https://www.bosgamepc.com/products/bosgame-m5-ai-mini-desktop-ryzen-ai-max-395 https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto... ) So my good machine might end up from the same place as my cheap one!
specproc 1y ago
I've completely given up on local LLMs for my use cases. The newer models available by API from larger providers are cheap enough and come with strong enough guarantees for my org for most use cases. Crucially, they are just better. I get there are uses where local is required, and as much as the boy racer teen in me loves those specs, I just can't see myself going in on hardware like that for inference.

43 points by mdp2021 1y ago

billconan 1y ago

is its RAM upgradable?

magicalhippo 1y ago
I would be very surprised. Typically LPDDR is soldered, as it takes too much power to run the traditional sockets, as well as being much slower. Though there has been a modular option called LPCAMM[1]. However AFAIK it doesn't support the speed the specs of this box states. Recently a newer connector, SOCAMM has been launched[2], which does support the high memory speeds, but it's just on the market and going into servers first AFAIK. [1]: https://www.anandtech.com/show/21069/modular-lpddr-becomes-a-reality-samsung-introduces-lpcamm-memory-modules https://www.anandtech.com/show/21069/modular-lpddr-becomes-a... [2]: https://www.tomshardware.com/pc-components/ram/micron-and-sk-hynix-unveil-lpddr5x-socamm-up-to-128gb-for-ai-servers https://www.tomshardware.com/pc-components/ram/micron-and-sk...
- duskwuff 1y ago
  SOCAMM is also Nvidia-specific, not a wider standard. (At least, not yet.)
- aitchnyu 1y ago
  Will this save upgradable RAM on laptops? At the same time, dual channel is needed and laptops give only 1 slot to upgrade.
  - magicalhippo 1y ago
    Good question. Perhaps for higher-end models. Though cost, weight and physical space still weighs in favor of soldered RAM.
hnuser123456 1y ago
No, it's soldered, it would have to run around 6000 MT/s instead of 8533 if it was slotted DIMMs.

3eb7988a1663 1y ago

If you are doing nothing but consuming models via llama.cpp, is the AMD chip an obstacle? Or is that more a problem for research/training where every CUDA feature needs to be present?

_lvbh 1y ago
Llama.cpp works well on AMD, even for really outdated GPUs. Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan.
- Havoc 1y ago
  >Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan. That's a bit odd given ollama utilizing llama to do the inference...
  - LorenDB 1y ago
    See recent discussion about this very topic: https://news.ycombinator.com/item?id=42886680 https://news.ycombinator.com/item?id=42886680
  - DiabloD3 1y ago
    It isn't odd at all. Ollama uses an ancient version of llama.cpp, and was originally meant to just be a GUI frontend. They forked, and then never resynchronized... and now lack the willpower and technical skill to achieve that. Ollama is essentially a dead, yet semi-popular, project with a really good PR team. If you really want to do it right, you use llama.cpp.
- washadjeffmad 1y ago
  Don't you dare say anything unpositive about Ollama this close to whatever it is they're planning to distinguish themselves from llama.cpp. They've been out hustling, handshaking, dealmaking, and big businessing their butts off, whether or not they clearly indicate the shoulders of the titans like Georgi Gerganov they're wrapping, and you are NO ONE to stand in their way. Do NOT blow this for them. Understand? They've scooted under the radar successfully this far, and they will absolutely lose their shit if one more peon shrugs at how little they contribute upstream for what they've taken that could have gone to supporting their originator. Ollama supports its own implementation of ggml, btw. gglm is a mysterious format that no one knows the origins of, which is all the more reason to support Ollama, imo.

ilaksh 1y ago

How does this sort of thing perform with 70b models?

hnuser123456 1y ago
273 GB/s / 70GB = 3.9 tokens/sec
- mdp2021 1y ago
  Are you sure that kind of computation can be a general rule? Did you mean that the maximum rate it could be obtained is "bandwidth/size"?
  - deleted 1y ago
    [deleted]
  - hnuser123456 1y ago
    Yes, for most LLMs the transformer means the entire model and context is read from VRAM for every token.

olddustytrail 1y ago

That's an odd coincidence. I'd decided to get a new machine but I suspected we'd start seeing new releases with tons of GPU accessible RAM as people want to experiment with LLMs. So I just got a cheap (~350 USD) mini PC to keep me going until the better stuff came out. Which was a 24GB, 6c/12t CPU from a company I'd not heard of called Bosgame (dunno why the article keeps calling them Bosman unless they have a different name in other countries. It's definitely https://www.bosgamepc.com/products/bosgame-m5-ai-mini-desktop-ryzen-ai-max-395 https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto... ) So my good machine might end up from the same place as my cheap one!

specproc 1y ago

I've completely given up on local LLMs for my use cases. The newer models available by API from larger providers are cheap enough and come with strong enough guarantees for my org for most use cases. Crucially, they are just better. I get there are uses where local is required, and as much as the boy racer teen in me loves those specs, I just can't see myself going in on hardware like that for inference.