7 ms·

It's looking like we'll have Chinese OSS to thank for being able to host our own intelligence, free from the whims of proprietary megacorps. I know it doesn't

by mythz 8mo ago

It's looking like we'll have Chinese OSS to thank for being able to host our own intelligence, free from the whims of proprietary megacorps. I know it doesn't make financial sense to self-host given how cheap OSS inference APIs are now, but it's comforting not being beholden to anyone or requiring a persistent internet connection for on-premise intelligence. Didn't expect to go back to macOS but they're basically the only feasible consumer option for running large models locally.

mikrl 8mo ago
>I know it doesn't make financial sense to self-host given how cheap OSS inference APIs are now You can calculate the exact cost of home inference, given you know your hardware and can measure electrical consumption and compare it to your bill. I have no idea what cloud inference in aggregate actually costs, whether it’s profitable or a VC infused loss leader that will spike in price later. That’s why I’m using cloud inference now to build out my local stack.
- mythz 8mo ago
  Not concerned with electricity cost - I have solar + battery with excess supply where most goes back to the grid for $0 compensation (AU special). But I did the napkin math on M3 Ultra ROI when DeepSeek V3 launched: at $0.70/2M tokens and 30 tps, a $10K M3 Ultra would take ~30 years of non-stop inference to break even - without even factoring in electricity. You clearly don't self-host to save money. You do it to own your intelligence, keep your privacy, and not be reliant on a persistent internet connection.
NiloCK 8mo ago
> Didn't expect to go back to macOS but their basically the only feasible consumer option for running large models locally. I presume here you are referring to running on the device in your lap. How about a headless linux inference box in the closet / basement? Return of the home network!
- Aurornis 8mo ago
  Apple devices have high memory bandwidth necessary to run LLMs at reasonable rates. It’s possible to build a Linux box that does the same but you’ll be spending a lot more to get there. With Apple, a $500 Mac Mini has memory bandwidth that you just can’t get anywhere else for the price.
  - cmrdporcupine 8mo ago
    But a $500 Mac Mini has nowhere near the memory capacity to run such a model. You'd need at least 2 512GB machines chained together to run this model. Maybe 1 if you quantized the crap out of it. And Apple completely overcharges for memory, so. This is a model you use via a cheap API provider like DeepInfra, or get on their coding plan. It's nice that it will be available as open weights, but not practical for mere mortals to run. But I can see a large corporation that wants to avoid sending code offsite setting up their own private infra to host it.
    - zozbot234 8mo ago
      The needed memory capacity depends on active parameters (not the same as total with a MoE model) and context length for the purpose of KV caching. Even then the KV cache can be pushed to system RAM and even farther out to swap, since writes to it are small (just one KV vector per token).
  - zozbot234 8mo ago
    And then only Apple devices have 512GB of unified memory, which matters when you have to combine larger models (even MoE) with the bigger context/KV caching you need for agentic workflows. You can make do with less, but only by slowing things down a whole lot.
  - ingenieroariel 8mo ago
    With Apple devices you get very fast predictions once it gets going but it is inferior to nvidia precisely during prefetch (processing prompt/context) before it really gets going. For our code assistant use cases the local inference on Macs will tend to favor workflows where there is a lot of generation and little reading and this is the opposite of how many of use use Claude Code. Source: I started getting Mac Studios with max ram as soon as the first llama model was released.
swalsh 8mo ago
Yeah that sounds great until it's running as an autonomous moltbot in a distributed network semi-offline with access to your entire digital life, and China sneaks in some hidden training so these agents turn into an army of sleeper agents.
- jfaat 8mo ago
  Lol wat? I mean you certainly have enough control self hosting the model to not let it join some moltbot network... or what exactly are you saying would happen?
  - swalsh 8mo ago
    We just saw last week people are setting up moltbots with virtually no knowledge of what it has and doesn't have access. The scenario that i'm afraid of is China realizes the potential of this. They can add training to the models commonly used for assistants. They act normal, are helpful, everything you'd want a bot to do. But maybe once in a while it checks moltbook or some other endpoint China controls for a trigger word. When it sees that, it kicks into a completely different mode, maybe it writes a script to DDoS targets of interest, maybe it mines your email for useful information, maybe the user has credentials to some piece that is a critical component of an important supply chain. This is not a wild scenario, no new sci-fi technology would need to be invented. Everything to do it is available today, people are configuring it, and using it like this today. The part that I fear is if it is running locally, you can't just shut off API access and kill the threat. It's running on it's own server, it's own model. You have to cut off each node. Big fan of AI, I use local models A LOT. I do think we have to take threats like this seriously. I don't Think it's a wild scifi idea. Since WW2, civilians have been as much of an equal opportunity target as a soldier, war is about logistics, and civilians supply the military.
    - resters 8mo ago
      Fair point but I would be more worried about the US government doing this kind of thing to act against US citizens than the Chinese government doing it. I think we're in a brief period of relative freedom where deep engineering topics can be discussed with AI agents even though they have potential uses in weapons systems. Imagine asking chat gpt how to build a fertilizer bomb, but apply the same censorship to anything related to computer vision, lasers, drone coordination, etc.
Aurornis 8mo ago
> It's looking like we'll have Chinese OSS to thank for being able to host our own intelligence, free from the whims of proprietary megacorps. I don’t know where you draw the line between proprietary megacorp and not, but Z.ai is planning to IPO soon as a multi billion dollar company. If you think they don’t want to be a multi billion dollar megacorp like all of the other LLM companies I think that’s a little short sighted. These models are open weight, but I wouldn’t count them as OSS. Also Chinese companies aren’t the only companies releasing open weight models. ChatGPT has released open weight models, too.
- joshstrange 8mo ago
  > Also Chinese companies aren’t the only companies releasing open weight models. ChatGPT has released open weight models, too. I was with you until here. The scraps OpenAI has released don't really compare to the GLM models or DeepSeek models (or others) in both cadence and quality (IMHO).
gz5 8mo ago
hopefully it will spread - many open options, from many entities, globally. it is brilliant business strategy from China so i expect it to continue and be copied - good things. reminds me of Google's investments into K8s.
btbuildem 8mo ago
> doesn't make financial sense to self-host I guess that's debatable. I regularly run out of quota on my claude max subscription. When that happens, I can sort of kind of get by with my modest setup (2x RTX3090) and quantized Qwen3. And this does not even account for privacy and availability. I'm in Canada, and as the US is slowly consumed by its spiral of self-destruction, I fully expect at some point a digital iron curtain will go up. I think it's prudent to have alternatives, especially with these paradigm-shattering tools.
- deleted 8mo ago
  [deleted]
- jsheard 8mo ago
  I think AI may be the only place you could get away with calling a 2x350W GPU rig "modest". That's like ten normal computers worth of power for the GPUs alone.
  - kataklasm 8mo ago
    Did you even try to read and understand the parent comment? They said they regularly run out of quota on the exact subscription you're advising they subscribe to.
    - h3half 8mo ago
      Pot, kettle
  - dymk 8mo ago
    That's maybe a few dollars to tens of dollars in electricity per month depending on where in the US you live
    - GeorgeOldfield 8mo ago
      the upfront cost
  - 8mo ago
nialv7 8mo ago
> Didn't expect to go back to macOS but their basically the only feasible consumer option for running large models locally. Framework Desktop! Half the memory bandwidth of M4 Max, but much cheaper.
- thebruce87m 8mo ago
  Does that equate to half the speed in terms of output? Any recommended benchmarks to look at?
  - nialv7 8mo ago
    https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/
throwaw12 8mo ago
our laptops, devices, phones, equipments, home stuff are all powered by Chinese companies. It wouldn't surprise me if at some point in the future my local "Alexa" assistant will be fully powered by local Chinese OSS models with Chinese GPUs and RAM.
andersa 8mo ago
They haven't published the weights yet, don't celebrate too early.
- andersa 8mo ago
  Now they have!
vidarh 8mo ago
I don't really care about being able to self host these models, but getting to a point where the hosting is commoditised so I know I can switch providers on a whim matters a great deal. Of course, it's nice if I can run it myself as a last resort too.
- RALaBarge 8mo ago
  It is pretty easy to set up Open Router and set up schemes to point at different models, but in the same token, you can point at yours locally unless you wanted a "more powerful" answer
fauigerzigerk 8mo ago
>...free from the whims of proprietary megacorps In one sense yes, but the training data is not open, nor is the data selection criteria (inclusions/exclusions, censorship, safety, etc). So we are still subject to the whims of someone much more powerful that ourselves. The good thing is that open weights models can be finetuned to correct any biases that we may find.
TheRealPomax 8mo ago
Not going to call $30/mo for a github copilot subscription "cheap". More like "extortionary".
- cmrdporcupine 8mo ago
  Yeah it's funny how the needle has moved on this kind of thing. Two years ago people scoffed at buying a personal license for e.g. JetBrains IDEs which netted out to $120 USD or something a year; VS Code etc took off because they were "free" But now they're dumping monthly subs to OpenAI and Anthropic that work out to the same as their car insurance payments. It's not sustainable.
  - TheRealPomax 8mo ago
    There's also zero incentive for individual companies to care: if I only want to use opus in VS code (and why would I use anything else, it's so much better at the job) I can either pay for copilot, which has excellent VS Code integration (because it has to), or I can pay Claude specifically and then use their extension which has the absolute worst experience because not only is the chat "whimsical, to make AI fun!", its interface is pat of the sidebar, so it's mutually exclusive with your file browser, search, etc. So whether you pay Claude or GitHub, Claude gets paid the same. So the consumer ends up footing a bill that has no reason to exist, and has no real competition because open source models can't run at the scale of an Opus or ChatGPT. (not unless the EU decides it's time for a "European Open AI Initiative" where any EU citizen gets free access to an EU wide datacenter backed large scale system that AI companies can pay to be part of, instead of getting paid to connect to)
mminer237 8mo ago
I'm not sure being beholden to the whims of the Chinese Communist Party is an iota better than the whims of proprietary megacorps, especially given this probably will become part of a megacorp anyway.
- hnfong 8mo ago
  It seems you missed the point entirely once you saw the word "Chinese". The point isn't that the models are from China. It's that the weights are open. You can download the weights and finetune them yourself. Nobody is beholden to anything.
  - mminer237 8mo ago
    Finetuning the weights doesn't eliminate bias though. Just seems like a bandaid.
deleted 8mo ago
[deleted]
muyuu 8mo ago
you have 128GB strix halo machines for US$ ~3k these run some pretty decent models locally, currently I'd recommend GPT-OSS 120GB, Qwen Coder Next 80B (either Q8 or Q6 quants, depending on speed/quality trade-offs) and the very best model you can run right now which is Step 3.5 Flash (ubergarm GGUF quant) with 256K context although this does push it to the limit - GLMs and nemotrons also worth trying depending on your priorities there's clearly a big quantum leap in the SotA models using more than 512GB VRAM, but i expect that in a year or two, the current SotA is achievable with consumer level hardware, if nothing else hardware should catch up with running Kimi 2.5 for cheaper than 2x 512GB mac studio ultras - perhaps medusa halo next year supports 512GB and DDR5 comes down again, and that would put a local whatever the best open model of that size is next year within reach of under-US$5K hardware the odd thing is that there isn't much in this whole range between 128GB and 512GB VRAM requirement to justify the huge premium you pay for Macs in that range - but this can change at any point as every other day there are announcements
- saubeidl 8mo ago
  And you can get Strix Halo in a Laptop that looks and feels like a Macbook Pro that can run Linux if you buy an HP ZBook G1A. Super happy with that thing, only real downside is battery life.
pzo 8mo ago
AFAIK they haven't released this one as OSS yet. They might eventually but its pretty obvious to me that at one point all/most those more powerful chinese models probably will stop being OSS.

by mythz 8mo ago

mikrl 8mo ago

>I know it doesn't make financial sense to self-host given how cheap OSS inference APIs are now You can calculate the exact cost of home inference, given you know your hardware and can measure electrical consumption and compare it to your bill. I have no idea what cloud inference in aggregate actually costs, whether it’s profitable or a VC infused loss leader that will spike in price later. That’s why I’m using cloud inference now to build out my local stack.

mythz 8mo ago
Not concerned with electricity cost - I have solar + battery with excess supply where most goes back to the grid for $0 compensation (AU special). But I did the napkin math on M3 Ultra ROI when DeepSeek V3 launched: at $0.70/2M tokens and 30 tps, a $10K M3 Ultra would take ~30 years of non-stop inference to break even - without even factoring in electricity. You clearly don't self-host to save money. You do it to own your intelligence, keep your privacy, and not be reliant on a persistent internet connection.

NiloCK 8mo ago

> Didn't expect to go back to macOS but their basically the only feasible consumer option for running large models locally. I presume here you are referring to running on the device in your lap. How about a headless linux inference box in the closet / basement? Return of the home network!

Aurornis 8mo ago
Apple devices have high memory bandwidth necessary to run LLMs at reasonable rates. It’s possible to build a Linux box that does the same but you’ll be spending a lot more to get there. With Apple, a $500 Mac Mini has memory bandwidth that you just can’t get anywhere else for the price.
- cmrdporcupine 8mo ago
  But a $500 Mac Mini has nowhere near the memory capacity to run such a model. You'd need at least 2 512GB machines chained together to run this model. Maybe 1 if you quantized the crap out of it. And Apple completely overcharges for memory, so. This is a model you use via a cheap API provider like DeepInfra, or get on their coding plan. It's nice that it will be available as open weights, but not practical for mere mortals to run. But I can see a large corporation that wants to avoid sending code offsite setting up their own private infra to host it.
  - zozbot234 8mo ago
    The needed memory capacity depends on active parameters (not the same as total with a MoE model) and context length for the purpose of KV caching. Even then the KV cache can be pushed to system RAM and even farther out to swap, since writes to it are small (just one KV vector per token).
- zozbot234 8mo ago
  And then only Apple devices have 512GB of unified memory, which matters when you have to combine larger models (even MoE) with the bigger context/KV caching you need for agentic workflows. You can make do with less, but only by slowing things down a whole lot.
- ingenieroariel 8mo ago
  With Apple devices you get very fast predictions once it gets going but it is inferior to nvidia precisely during prefetch (processing prompt/context) before it really gets going. For our code assistant use cases the local inference on Macs will tend to favor workflows where there is a lot of generation and little reading and this is the opposite of how many of use use Claude Code. Source: I started getting Mac Studios with max ram as soon as the first llama model was released.

swalsh 8mo ago

Yeah that sounds great until it's running as an autonomous moltbot in a distributed network semi-offline with access to your entire digital life, and China sneaks in some hidden training so these agents turn into an army of sleeper agents.

jfaat 8mo ago
Lol wat? I mean you certainly have enough control self hosting the model to not let it join some moltbot network... or what exactly are you saying would happen?
- swalsh 8mo ago
  We just saw last week people are setting up moltbots with virtually no knowledge of what it has and doesn't have access. The scenario that i'm afraid of is China realizes the potential of this. They can add training to the models commonly used for assistants. They act normal, are helpful, everything you'd want a bot to do. But maybe once in a while it checks moltbook or some other endpoint China controls for a trigger word. When it sees that, it kicks into a completely different mode, maybe it writes a script to DDoS targets of interest, maybe it mines your email for useful information, maybe the user has credentials to some piece that is a critical component of an important supply chain. This is not a wild scenario, no new sci-fi technology would need to be invented. Everything to do it is available today, people are configuring it, and using it like this today. The part that I fear is if it is running locally, you can't just shut off API access and kill the threat. It's running on it's own server, it's own model. You have to cut off each node. Big fan of AI, I use local models A LOT. I do think we have to take threats like this seriously. I don't Think it's a wild scifi idea. Since WW2, civilians have been as much of an equal opportunity target as a soldier, war is about logistics, and civilians supply the military.
  - resters 8mo ago
    Fair point but I would be more worried about the US government doing this kind of thing to act against US citizens than the Chinese government doing it. I think we're in a brief period of relative freedom where deep engineering topics can be discussed with AI agents even though they have potential uses in weapons systems. Imagine asking chat gpt how to build a fertilizer bomb, but apply the same censorship to anything related to computer vision, lasers, drone coordination, etc.

Aurornis 8mo ago

> It's looking like we'll have Chinese OSS to thank for being able to host our own intelligence, free from the whims of proprietary megacorps. I don’t know where you draw the line between proprietary megacorp and not, but Z.ai is planning to IPO soon as a multi billion dollar company. If you think they don’t want to be a multi billion dollar megacorp like all of the other LLM companies I think that’s a little short sighted. These models are open weight, but I wouldn’t count them as OSS. Also Chinese companies aren’t the only companies releasing open weight models. ChatGPT has released open weight models, too.

joshstrange 8mo ago
> Also Chinese companies aren’t the only companies releasing open weight models. ChatGPT has released open weight models, too. I was with you until here. The scraps OpenAI has released don't really compare to the GLM models or DeepSeek models (or others) in both cadence and quality (IMHO).

gz5 8mo ago

hopefully it will spread - many open options, from many entities, globally. it is brilliant business strategy from China so i expect it to continue and be copied - good things. reminds me of Google's investments into K8s.

btbuildem 8mo ago

> doesn't make financial sense to self-host I guess that's debatable. I regularly run out of quota on my claude max subscription. When that happens, I can sort of kind of get by with my modest setup (2x RTX3090) and quantized Qwen3. And this does not even account for privacy and availability. I'm in Canada, and as the US is slowly consumed by its spiral of self-destruction, I fully expect at some point a digital iron curtain will go up. I think it's prudent to have alternatives, especially with these paradigm-shattering tools.

deleted 8mo ago
[deleted]
jsheard 8mo ago
I think AI may be the only place you could get away with calling a 2x350W GPU rig "modest". That's like ten normal computers worth of power for the GPUs alone.
- kataklasm 8mo ago
  Did you even try to read and understand the parent comment? They said they regularly run out of quota on the exact subscription you're advising they subscribe to.
  - h3half 8mo ago
    Pot, kettle
- dymk 8mo ago
  That's maybe a few dollars to tens of dollars in electricity per month depending on where in the US you live
  - GeorgeOldfield 8mo ago
    the upfront cost
- 8mo ago

nialv7 8mo ago

> Didn't expect to go back to macOS but their basically the only feasible consumer option for running large models locally. Framework Desktop! Half the memory bandwidth of M4 Max, but much cheaper.

thebruce87m 8mo ago
Does that equate to half the speed in terms of output? Any recommended benchmarks to look at?
- nialv7 8mo ago
  https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/

throwaw12 8mo ago

our laptops, devices, phones, equipments, home stuff are all powered by Chinese companies. It wouldn't surprise me if at some point in the future my local "Alexa" assistant will be fully powered by local Chinese OSS models with Chinese GPUs and RAM.

andersa 8mo ago

They haven't published the weights yet, don't celebrate too early.

andersa 8mo ago
Now they have!

vidarh 8mo ago

I don't really care about being able to self host these models, but getting to a point where the hosting is commoditised so I know I can switch providers on a whim matters a great deal. Of course, it's nice if I can run it myself as a last resort too.

RALaBarge 8mo ago
It is pretty easy to set up Open Router and set up schemes to point at different models, but in the same token, you can point at yours locally unless you wanted a "more powerful" answer

fauigerzigerk 8mo ago

>...free from the whims of proprietary megacorps In one sense yes, but the training data is not open, nor is the data selection criteria (inclusions/exclusions, censorship, safety, etc). So we are still subject to the whims of someone much more powerful that ourselves. The good thing is that open weights models can be finetuned to correct any biases that we may find.

TheRealPomax 8mo ago

Not going to call $30/mo for a github copilot subscription "cheap". More like "extortionary".

cmrdporcupine 8mo ago
Yeah it's funny how the needle has moved on this kind of thing. Two years ago people scoffed at buying a personal license for e.g. JetBrains IDEs which netted out to $120 USD or something a year; VS Code etc took off because they were "free" But now they're dumping monthly subs to OpenAI and Anthropic that work out to the same as their car insurance payments. It's not sustainable.
- TheRealPomax 8mo ago
  There's also zero incentive for individual companies to care: if I only want to use opus in VS code (and why would I use anything else, it's so much better at the job) I can either pay for copilot, which has excellent VS Code integration (because it has to), or I can pay Claude specifically and then use their extension which has the absolute worst experience because not only is the chat "whimsical, to make AI fun!", its interface is pat of the sidebar, so it's mutually exclusive with your file browser, search, etc. So whether you pay Claude or GitHub, Claude gets paid the same. So the consumer ends up footing a bill that has no reason to exist, and has no real competition because open source models can't run at the scale of an Opus or ChatGPT. (not unless the EU decides it's time for a "European Open AI Initiative" where any EU citizen gets free access to an EU wide datacenter backed large scale system that AI companies can pay to be part of, instead of getting paid to connect to)

mminer237 8mo ago

I'm not sure being beholden to the whims of the Chinese Communist Party is an iota better than the whims of proprietary megacorps, especially given this probably will become part of a megacorp anyway.

hnfong 8mo ago
It seems you missed the point entirely once you saw the word "Chinese". The point isn't that the models are from China. It's that the weights are open. You can download the weights and finetune them yourself. Nobody is beholden to anything.
- mminer237 8mo ago
  Finetuning the weights doesn't eliminate bias though. Just seems like a bandaid.

deleted 8mo ago

[deleted]

muyuu 8mo ago

you have 128GB strix halo machines for US$ ~3k these run some pretty decent models locally, currently I'd recommend GPT-OSS 120GB, Qwen Coder Next 80B (either Q8 or Q6 quants, depending on speed/quality trade-offs) and the very best model you can run right now which is Step 3.5 Flash (ubergarm GGUF quant) with 256K context although this does push it to the limit - GLMs and nemotrons also worth trying depending on your priorities there's clearly a big quantum leap in the SotA models using more than 512GB VRAM, but i expect that in a year or two, the current SotA is achievable with consumer level hardware, if nothing else hardware should catch up with running Kimi 2.5 for cheaper than 2x 512GB mac studio ultras - perhaps medusa halo next year supports 512GB and DDR5 comes down again, and that would put a local whatever the best open model of that size is next year within reach of under-US$5K hardware the odd thing is that there isn't much in this whole range between 128GB and 512GB VRAM requirement to justify the huge premium you pay for Macs in that range - but this can change at any point as every other day there are announcements

saubeidl 8mo ago
And you can get Strix Halo in a Laptop that looks and feels like a Macbook Pro that can run Linux if you buy an HP ZBook G1A. Super happy with that thing, only real downside is battery life.

pzo 8mo ago

AFAIK they haven't released this one as OSS yet. They might eventually but its pretty obvious to me that at one point all/most those more powerful chinese models probably will stop being OSS.