5 ms·

what is the pitch, because desktops don't have a problem with replacements, repairs etc.

by calvinmorrison 2y ago

sunshowers 2y ago
256 GB/s memory bandwidth for $2000.
- bryanlarsen 2y ago
  Which should also be available from all the usual motherboard manufacturers. Possibly well before this one, since it doesn't ship until Q3.
  - sunshowers 2y ago
    Will it? I'm not aware of any other than the HP workstation. Maybe one or two of the Chinese mini PC manufacturers. But nothing you can buy as a mini-ITX motherboard.
    - bryanlarsen 2y ago
      Of course you will. Asus et al have heard the buzz, have the connections, and can spin up a product far quicker than the 5-9 or months before Strix Halo is available.
      - sunshowers 2y ago
        Sure, to the extent that happens, the Framework will compete with them. Competition is great.
  - zamalek 2y ago
    It's soldered RAM, which is a fantastic trade-off for specific scenarios (inference). If all you care about is local inference, this thing is basically the same price as a 5090 (I think?) with multiple times the memory, and no need to purchase "everything else" (mobo, CPU, etc.) alongside. And given that home inference will typically be serving a single user at once, a handful at worst, you really have no need for a GPU. I'm guessing that this product will be uniquely positioned for quite a long time. For every other use-case? Yeah, just get a desktop.
lytedev 2y ago
It seems to be squaring up directly against the mac studio with its efficient APU and big memory bandwidth use cases with a cheaper price tag. At least that's the loose sense that I got based on their keynote.
- jackbravo 2y ago
  Nvidia project Digits seems to fall in a similar category, no?
  - sunshowers 2y ago
    It is, and the keynote briefly mentioned it.
  - theossuary 2y ago
    It's a bit different. Digits is based on the Tegra CPU, which is an ARM chip with integrated nvidia GPU. It's nearly COTS (commercial off the shelf), but not quite. Tegra CPU support isn't in mainline linux, so you have to run their fork of Ubuntu or build your own kernel. The integrated GPU is a special class in nvidia drivers, and some things just don't work on it (they only work on a discrete GPU) for seemingly no reason too.
    - sliken 2y ago
      Keep in mind the digits is part of their server line and server OS, not the random embedded dev kits often used for developers targeting car entertainment systems and often with abandoned kernel + driver. The nvidia digits uses DGOS, same as their grace+hopper and similar enterprise/cloud products.
    - baobabKoodaa 2y ago
      Can you expand on this? If I want to run open-weight LLMs and image generation models in the next few years, how likely is it that I will be able to run the "most popular" models (whatever they will be 12 months from now) if I buy Digits?
  - 2y ago
Nifty3929 2y ago
Right - there was no major market gap here. With laptops there was, but not desktops. Not sure the point of this. I hope they didn't spend too much eng time on it, rather than on their laptops. The F16 could use a new rev...
dvtkrlbs 2y ago
128GB unified memory
boricj 2y ago
If that was available when I started to build my homelab server, I'd have bought it. My requirements were a low-power but modern and punchy mini-ITX board with an AMD processor in a very compact build with a 48v DC power supply and SmartOS. That was basically unobtainium and I've compromised down to a AliExpress mini-ITX motherboard with a mobile AMD CPU, a AM5 heatsink and firmware that is... flavorful, powered by a screaming TFX power supply crammed in an absurdly tight 3.8L noname case running on Proxmox (when you start reconfiguring PCI Express bridges through the serial port kernel debugger because that's just about the only device the Illumos kernel enumerated at all on what is supposed to be your main server, it's time to give up on SmartOS). It works, but it's a little box of pure hatred and heresy that's quite far off from what I've wanted initially. It is actually an improvement over my previous main server, somehow.
- dangus 2y ago
  No offense but your requirements make very little sense for that use case, unless you really needed PCI-Express. I’d have bought a Beelink or similar mini PC if I wanted small size and low power along with low price. You lose some modularity compared to ITX boards but I am almost certain you spent more money and deal with more noise and maybe even more power consumption. For me personally my homelab PC is just an ATX mid tower in the closet because those parts are dirt cheap and you can get lots of performance with essentially infinite modularity.
  - boricj 2y ago
    Originally, the homelab was supposed to be located inside a 6U, 30cm deep, encased 19" rack. That would fit flush and very snugly just below my encased electrical panel, located right at the entrance inside my apartment. That makes noise (a glass panel separates the rack from the entrance), power (can't risk overheating in an enclosed closet) and space (6U and 25cm of usable depth is an exceedingly small volume for a homelab) all important factors at the same time. I also wanted at least 2.5 Gib Ethernet, as much DDR5 RAM and CPU cores as I could chuck into it and two NVMe slots. I rejected a mini PC due to noise concerns and I wanted standardized parts and modularity. I have a AMD Ryzen 9 7940HS, 64 GiB of RAM and terabytes of NVMe in less than 4 liters of volume without sacrificing modularity. I've mounted an AMD Wraith Stealth, which is a comically oversized CPU cooling fan for a 35W TDP thermal load. It's fast, small, silent, overkill and still fits standard parts. I do want to get rid of the ridiculously noisy TFX power supply at some point, most likely for a 48v DC ATX power supply. Unfortunately, since I like stupid ideas I want a fully 48v DC powered rack and Power-over-Ethernet throughout my apartment to get rid of as many power supplies and fans as I can. I've started accumulating parts for it (like a 48v 2.5 Gib 10 port PoE switch, PoE powered gigabit switches and PoE-to-12VDC adapters), but I'm still not sure how I'll pull it off in the end since I'm way off the beaten path. At this point you're probably wondering why I do things that way. I'm not sure myself, but seeing what I tend to do for fun it's probably best not to ask, lest the universe stops its suspension of disbelief while I'm in the middle of performing yet another heretical act on some poor unsuspecting piece of hardware or software.
h14h 2y ago
Repair-friendly form factor for a "Unified Memory" platform. For $2000, you get 128 of system RAM, 96 of which is addressable as VRAM. Only ways of getting 96GB of VRAM in a desktop are to either: 1. Drop ~$5000 on a (very non-upgradeable) Mac Studio 2. Drop ~$20k on a dual RTX 6000 workstation For running local LLMs, there's nothing on the market presently even remotely like this.
- deleted 2y ago
  [deleted]
- seanmcdirmid 2y ago
  You can get a M3/M4 Max with 128 GB of RAM as well. The Studio will give you > 128 GB. I have a max with 64GB RAM, which is good enough for 70b models with a 3 bit quant. Even if I had more RAM to run larger models, my GPU would be the bottleneck.
  - Dylan16807 2y ago
    > You can get a M3/M4 Max with 128 GB of RAM as well. To get an M4 Max so you can have 128GB, you need a macbook pro. The cheapest macbook pro with 128GB is $4700. M4 Max does have the benefit of more memory controllers, so it has twice as much memory bandwidth as Ryzen AI Max. But that's a lot of money to pay for it.
    - orangecat 2y ago
      To get an M4 Max so you can have 128GB, you need a macbook pro. By the time the Framework ships the Mac Studio will have been updated to the M4 Max. Although 128GB will still probably be around $3k.
    - seanmcdirmid 2y ago
      Yep. You can get an M3 Max refurbished, but will make some tradeoff with GPU performance.

what is the pitch, because desktops don't have a problem with replacements, repairs etc.

sunshowers 2y ago

256 GB/s memory bandwidth for $2000.

bryanlarsen 2y ago
Which should also be available from all the usual motherboard manufacturers. Possibly well before this one, since it doesn't ship until Q3.
- sunshowers 2y ago
  Will it? I'm not aware of any other than the HP workstation. Maybe one or two of the Chinese mini PC manufacturers. But nothing you can buy as a mini-ITX motherboard.
  - bryanlarsen 2y ago
    Of course you will. Asus et al have heard the buzz, have the connections, and can spin up a product far quicker than the 5-9 or months before Strix Halo is available.
    - sunshowers 2y ago
      Sure, to the extent that happens, the Framework will compete with them. Competition is great.
- zamalek 2y ago
  It's soldered RAM, which is a fantastic trade-off for specific scenarios (inference). If all you care about is local inference, this thing is basically the same price as a 5090 (I think?) with multiple times the memory, and no need to purchase "everything else" (mobo, CPU, etc.) alongside. And given that home inference will typically be serving a single user at once, a handful at worst, you really have no need for a GPU. I'm guessing that this product will be uniquely positioned for quite a long time. For every other use-case? Yeah, just get a desktop.

lytedev 2y ago

It seems to be squaring up directly against the mac studio with its efficient APU and big memory bandwidth use cases with a cheaper price tag. At least that's the loose sense that I got based on their keynote.

jackbravo 2y ago
Nvidia project Digits seems to fall in a similar category, no?
- sunshowers 2y ago
  It is, and the keynote briefly mentioned it.
- theossuary 2y ago
  It's a bit different. Digits is based on the Tegra CPU, which is an ARM chip with integrated nvidia GPU. It's nearly COTS (commercial off the shelf), but not quite. Tegra CPU support isn't in mainline linux, so you have to run their fork of Ubuntu or build your own kernel. The integrated GPU is a special class in nvidia drivers, and some things just don't work on it (they only work on a discrete GPU) for seemingly no reason too.
  - sliken 2y ago
    Keep in mind the digits is part of their server line and server OS, not the random embedded dev kits often used for developers targeting car entertainment systems and often with abandoned kernel + driver. The nvidia digits uses DGOS, same as their grace+hopper and similar enterprise/cloud products.
  - baobabKoodaa 2y ago
    Can you expand on this? If I want to run open-weight LLMs and image generation models in the next few years, how likely is it that I will be able to run the "most popular" models (whatever they will be 12 months from now) if I buy Digits?
- 2y ago

Nifty3929 2y ago

Right - there was no major market gap here. With laptops there was, but not desktops. Not sure the point of this. I hope they didn't spend too much eng time on it, rather than on their laptops. The F16 could use a new rev...

dvtkrlbs 2y ago

128GB unified memory

boricj 2y ago

If that was available when I started to build my homelab server, I'd have bought it. My requirements were a low-power but modern and punchy mini-ITX board with an AMD processor in a very compact build with a 48v DC power supply and SmartOS. That was basically unobtainium and I've compromised down to a AliExpress mini-ITX motherboard with a mobile AMD CPU, a AM5 heatsink and firmware that is... flavorful, powered by a screaming TFX power supply crammed in an absurdly tight 3.8L noname case running on Proxmox (when you start reconfiguring PCI Express bridges through the serial port kernel debugger because that's just about the only device the Illumos kernel enumerated at all on what is supposed to be your main server, it's time to give up on SmartOS). It works, but it's a little box of pure hatred and heresy that's quite far off from what I've wanted initially. It is actually an improvement over my previous main server, somehow.

dangus 2y ago
No offense but your requirements make very little sense for that use case, unless you really needed PCI-Express. I’d have bought a Beelink or similar mini PC if I wanted small size and low power along with low price. You lose some modularity compared to ITX boards but I am almost certain you spent more money and deal with more noise and maybe even more power consumption. For me personally my homelab PC is just an ATX mid tower in the closet because those parts are dirt cheap and you can get lots of performance with essentially infinite modularity.
- boricj 2y ago
  Originally, the homelab was supposed to be located inside a 6U, 30cm deep, encased 19" rack. That would fit flush and very snugly just below my encased electrical panel, located right at the entrance inside my apartment. That makes noise (a glass panel separates the rack from the entrance), power (can't risk overheating in an enclosed closet) and space (6U and 25cm of usable depth is an exceedingly small volume for a homelab) all important factors at the same time. I also wanted at least 2.5 Gib Ethernet, as much DDR5 RAM and CPU cores as I could chuck into it and two NVMe slots. I rejected a mini PC due to noise concerns and I wanted standardized parts and modularity. I have a AMD Ryzen 9 7940HS, 64 GiB of RAM and terabytes of NVMe in less than 4 liters of volume without sacrificing modularity. I've mounted an AMD Wraith Stealth, which is a comically oversized CPU cooling fan for a 35W TDP thermal load. It's fast, small, silent, overkill and still fits standard parts. I do want to get rid of the ridiculously noisy TFX power supply at some point, most likely for a 48v DC ATX power supply. Unfortunately, since I like stupid ideas I want a fully 48v DC powered rack and Power-over-Ethernet throughout my apartment to get rid of as many power supplies and fans as I can. I've started accumulating parts for it (like a 48v 2.5 Gib 10 port PoE switch, PoE powered gigabit switches and PoE-to-12VDC adapters), but I'm still not sure how I'll pull it off in the end since I'm way off the beaten path. At this point you're probably wondering why I do things that way. I'm not sure myself, but seeing what I tend to do for fun it's probably best not to ask, lest the universe stops its suspension of disbelief while I'm in the middle of performing yet another heretical act on some poor unsuspecting piece of hardware or software.

h14h 2y ago

Repair-friendly form factor for a "Unified Memory" platform. For $2000, you get 128 of system RAM, 96 of which is addressable as VRAM. Only ways of getting 96GB of VRAM in a desktop are to either: 1. Drop ~$5000 on a (very non-upgradeable) Mac Studio 2. Drop ~$20k on a dual RTX 6000 workstation For running local LLMs, there's nothing on the market presently even remotely like this.

deleted 2y ago
[deleted]
seanmcdirmid 2y ago
You can get a M3/M4 Max with 128 GB of RAM as well. The Studio will give you > 128 GB. I have a max with 64GB RAM, which is good enough for 70b models with a 3 bit quant. Even if I had more RAM to run larger models, my GPU would be the bottleneck.
- Dylan16807 2y ago
  > You can get a M3/M4 Max with 128 GB of RAM as well. To get an M4 Max so you can have 128GB, you need a macbook pro. The cheapest macbook pro with 128GB is $4700. M4 Max does have the benefit of more memory controllers, so it has twice as much memory bandwidth as Ryzen AI Max. But that's a lot of money to pay for it.
  - orangecat 2y ago
    To get an M4 Max so you can have 128GB, you need a macbook pro. By the time the Framework ships the Mac Studio will have been updated to the M4 Max. Although 128GB will still probably be around $3k.
  - seanmcdirmid 2y ago
    Yep. You can get an M3 Max refurbished, but will make some tradeoff with GPU performance.