3 ms·

Would it though? How many people are running inference at home? Outside of enthusiasts I don't know anyone. Even companies don't self-host models and prefer to

Would it though? How many people are running inference at home? Outside of enthusiasts I don't know anyone. Even companies don't self-host models and prefer to use APIs. Not that I wouldn't like a consumer GPU with tons of VRAM, but I think that the market for it is quite small for companies to invest building it. If you bother to look at Steam's hardware stats you'll notice that only a small percentage is using high-end cards.

ModernMech 2y ago
It's a chicken and egg scenario. The main problem with running inference at home is the lack of hardware. If the hardware was there more people would do it. And it's not a problem if "enthusiasts" are the only ones using it because that's to be expected at this stage of the tech cycle. If the market is small just charge more, the enthusiasts will pay it. Once more enthusiasts are running inference at home, then the late adopters will eventually come along.
- m00x 2y ago
  Mac minis are great for this. They're cheap-ish and they can run quite large models at a decent speed if you run it with an MLX backend.
  - alganet 2y ago
    mini _Pro_ are great for this, ones with large RAM upgrades. If you get the base 16GB mini, it will have more or less the same VRAM but way worse performance than an Arc. If you already have a PC, it makes sense to go for the cheapest 12GB card instead of a base mac mini.
tokioyoyo 2y ago
This is the weird part, I saw the same comments in other threads. People keep saying how everyone yearns for local LLMs… but other than hardcore enthusiasts it just sounds like a bad investment? Like it’s a smaller market than gaming GPUs. And by the time anyone runs them locally, you’ll have bigger/better models and GPUs coming out, so you won’t even be able to make use of them. Maybe the whole “indoctrinate users to be a part of Intel ecosystem, so when they go work for big companies they would vouch for it” would have merit… if others weren’t innovating and making their products better (like NVIDIA).
- throwaway48476 2y ago
  Intel sold their GPUs at negative margin which is part of why the stock fell off a cliff. If they could double the vram they could raise the price into the green even selling thousands, likely closer to 100k, would be far better than what they're doing now. The problem is Intel is run by incompetent people who guard their market segments as tribal fiefs instead of solving for the customer.
  - refulgentis 2y ago
    By subsidizing it more they'll lose less money?
    - throwaway48476 2y ago
      Increasing VRAM would differentiate intel GPUs and allow driving higher ASPs, into the green.
      - refulgentis 2y ago
        Well, IIUC it's a bit more "having more than 12GB of RAM and raising the price will let it run bigger LLMs on consumer hardware and that'll drive premium-ness / market share / revenue, without subsidizing the price" I don't know where this idea is coming from, although it's all over these threads. For context, I write a local LLM inference engine and have 0 idea why this would shift anyone's purchase intent. The models big enough to need more than 12GB VRAM are also slow enough on consumer GPUs that they'd be absurd to run. Like less than 2 tkns/s. And I have 64 GB of M2 Max VRAM and a 24 GB 3090ti.
      - kimixa 2y ago
        I think you're massively underestimating the development cost, of the number of people who would actually purchase a higher vram card at a higher price. You'd need hundreds of thousands of units to really make much of a difference.
  - qwytw 2y ago
JohnBooty 2y ago
Would it though? How many people are running inference at home? I don't know how to quantify it, but it certainly seems like a lot of people are buying consumer nVidia GPUs for compute and the relatively paltry amounts of RAM on those cards seems to be the number one complaint. So I would say that Intel's potential market is "everybody who is currently buying nVidia GPUs for compute." nVidia's stingy consumer RAM choices also seem to be a fairly transparent ploy to create a protective moat around their insanely-high-profit-margin datacenter GPUs. So that just seems like kind of an obvious thing for Intel or AMD to consider tackling. (Although, it has to be said, a lot of commenters have pointed out that it's not as easy as just slapping more RAM chips onto the GPU boards; you need wider data busses as well etc.)

ModernMech 2y ago

It's a chicken and egg scenario. The main problem with running inference at home is the lack of hardware. If the hardware was there more people would do it. And it's not a problem if "enthusiasts" are the only ones using it because that's to be expected at this stage of the tech cycle. If the market is small just charge more, the enthusiasts will pay it. Once more enthusiasts are running inference at home, then the late adopters will eventually come along.

m00x 2y ago
Mac minis are great for this. They're cheap-ish and they can run quite large models at a decent speed if you run it with an MLX backend.
- alganet 2y ago
  mini _Pro_ are great for this, ones with large RAM upgrades. If you get the base 16GB mini, it will have more or less the same VRAM but way worse performance than an Arc. If you already have a PC, it makes sense to go for the cheapest 12GB card instead of a base mac mini.

tokioyoyo 2y ago

This is the weird part, I saw the same comments in other threads. People keep saying how everyone yearns for local LLMs… but other than hardcore enthusiasts it just sounds like a bad investment? Like it’s a smaller market than gaming GPUs. And by the time anyone runs them locally, you’ll have bigger/better models and GPUs coming out, so you won’t even be able to make use of them. Maybe the whole “indoctrinate users to be a part of Intel ecosystem, so when they go work for big companies they would vouch for it” would have merit… if others weren’t innovating and making their products better (like NVIDIA).

throwaway48476 2y ago
Intel sold their GPUs at negative margin which is part of why the stock fell off a cliff. If they could double the vram they could raise the price into the green even selling thousands, likely closer to 100k, would be far better than what they're doing now. The problem is Intel is run by incompetent people who guard their market segments as tribal fiefs instead of solving for the customer.
- refulgentis 2y ago
  By subsidizing it more they'll lose less money?
  - throwaway48476 2y ago
    Increasing VRAM would differentiate intel GPUs and allow driving higher ASPs, into the green.
    - refulgentis 2y ago
      Well, IIUC it's a bit more "having more than 12GB of RAM and raising the price will let it run bigger LLMs on consumer hardware and that'll drive premium-ness / market share / revenue, without subsidizing the price" I don't know where this idea is coming from, although it's all over these threads. For context, I write a local LLM inference engine and have 0 idea why this would shift anyone's purchase intent. The models big enough to need more than 12GB VRAM are also slow enough on consumer GPUs that they'd be absurd to run. Like less than 2 tkns/s. And I have 64 GB of M2 Max VRAM and a 24 GB 3090ti.
    - kimixa 2y ago
      I think you're massively underestimating the development cost, of the number of people who would actually purchase a higher vram card at a higher price. You'd need hundreds of thousands of units to really make much of a difference.
- qwytw 2y ago

JohnBooty 2y ago

Would it though? How many people are running inference at home? I don't know how to quantify it, but it certainly seems like a lot of people are buying consumer nVidia GPUs for compute and the relatively paltry amounts of RAM on those cards seems to be the number one complaint. So I would say that Intel's potential market is "everybody who is currently buying nVidia GPUs for compute." nVidia's stingy consumer RAM choices also seem to be a fairly transparent ploy to create a protective moat around their insanely-high-profit-margin datacenter GPUs. So that just seems like kind of an obvious thing for Intel or AMD to consider tackling. (Although, it has to be said, a lot of commenters have pointed out that it's not as easy as just slapping more RAM chips onto the GPU boards; you need wider data busses as well etc.)