7 ms·

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on som

by fooker 3mo ago

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really really narrow. I wouldn't be surprised if this is an empty set.

walrus01 3mo ago
There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.
- fooker 3mo ago
  Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
  - walrus01 3mo ago
    Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output. On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.
  - sm-silversight 3mo ago
    >Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second? Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.
    - walrus01 3mo ago
      Very good description of it. It does seem like a bit of a rhetorical question to ask a forum that has a very high population of Linux and BSD users why they might desire to have the option to do something themselves rather than relying on an external packaged ready to go product.
  - irishcoffee 3mo ago
    The whole mentality of thinking one knows better than another about what they need causes infinitely more problems than it solves.
- light_hue_1 3mo ago
  As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.
  - walrus01 3mo ago
    Do you mean by trading dollars for the privacy you need as: a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place or b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place or c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds? Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.
    - jmalicki 3mo ago
      A) is very doable with e.g. Amazon Bedrock. They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights. What kind of privacy needs do you really have beyond that?
      - walrus01 3mo ago
        It is not my use case but given recent political developments in international relations caused by the executive branch of the US government, off the top of my head, I could think of a lot of European or Canadian firms for which that would not be an option. No matter what they might promise about European sovereignty. For a good 'ol patriotic US domestic company? Sure.
- mdasen 3mo ago
  That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents. There are parts of states like Grant County Washington that have cheap hydro power, but it's very rare for power to be that cheap in the US. Even if this applies to you, it won't apply to the vast majority of people on here who will have electric rates 2-4x higher. Average electric rates by region: New England 28.1 cents Mid Atlantic 25.1 cents East North Central 20.8 cents West North Central 14.8 cents South Atlantic 16.1 cents East South Central 15.5 cents Mountain 14.6 cents Pacific Contiguous 26.1 cents Pacific Noncontiguous 42.1 cents https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
  - peterleiser 3mo ago
    Nashville Electric Service rates are 12.0 cents / kWh. Flat rate, no peak or off peak rates for residential.
  - AgentMatt 3mo ago
    They are most likely not based in the US, but converting to USD to make comparison easier.
    - walrus01 3mo ago
      I am not in Quebec but Quebec hydro rate D for standard residential would be one example of around what I pay. https://www.hydroquebec.com/residential/customer-space/rates/rate-d.html https://www.hydroquebec.com/residential/customer-space/rates... Another example would be Manitoba hydro All figures in Canadian currency https://www.hydro.mb.ca/account/rates/residential/ https://www.hydro.mb.ca/account/rates/residential/
    - brailsafe 3mo ago
      Specifying USD is indeed often a service usually offered by people born elsewhere for people born elsewhere. Americans seem rarely know about these mysterious places, where bills can come in all sorts of funny sizes and colours. (kind of joking)
- ljlolel 3mo ago
  can send safely context if there’s confidential computing ala my site https://trustedrouter.com/ https://trustedrouter.com/
  - skinfaxi 3mo ago
    How do you prove you are running exclusively on Nitro enclave instances or GCP confidential spaces?
    - Cider9986 3mo ago
      Is your local compute airgapped?
      - well_ackshually 3mo ago
        I trust my local compute quite a bit more than a random project that has "trusted" in its name.
      - trollbridge 3mo ago
        My local compute is used by me, and I'm accountable to myself whether or not is secure. So to a certain degree, I trust myself and also know what limitations / potential vulnerabilities it might have.
    - whs 3mo ago
      Seems clear from their website? 1. Their API server provide an attestation JWT. This JWT is signed by Google's private key. 2. The attestation has details on the running container. I suppose the container host is a Google-provided distro and Google's signer will verify that the OS is theirs and up-to-date. 3. They could've proxy the attestation. To prove this is not the case, the field eat_nonce include the TLS certificate fingerprint, which should match the API server you're connecting to. I suppose you will need to pull their container and verify from the source that the container itself generate the private key, it never leaves the container, and the container has no way to run arbitrary code such as SSH or vulnerabilities.
- tiahura 3mo ago
  There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. Are there? At the highest levels of defense and law, AWS and Azure are used. Having tried selling some of these entities on doing things in-house, there seems to be little interest.
  - walrus01 3mo ago
    > Are there? At the highest levels of defense and law, AWS and Azure are used. This is certainly true if the user is an American company. You could look at the European initiatives to run this stuff on hardware they own in facilities they own and control within the borders of Europe for a counter-example. Such as: https://www.google.com/search?client=firefox-b-d&q=schwarz+stackit+germany https://www.google.com/search?client=firefox-b-d&q=schwarz+s... https://www.dutchnews.nl/2026/04/government-turns-to-german-cloud-provider-to-cut-us-reliance/ https://www.dutchnews.nl/2026/04/government-turns-to-german-...
    - jmalicki 3mo ago
      Yeah, true European cloud providers for these kinds of things seem to be behind, and a lot of the ones offering data compliance at the level of AWS are small enough that it's a bit harder to trust they'll be around and will keep their promises. Hopefully that changes!
- doctorpangloss 3mo ago
  It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.
  - trollbridge 3mo ago
    Yeah. At 5 tok/second, you're talking about around $195 worth of output tokens per month. There is no way I can run a usable K3 model for $195 a month of capex, opex, or any-kind-of-ex. Qwen 3.6 is another matter. Paying provider rates for the amount I run locally would put me in the thousands of dollars. So that's very practical to buy a Macbook instead, plus an RTX card, and so on.
- polycancel 3mo ago
  >and people will compromise speed for data sovereignty People should always compromise speed for data sovereignty! Who said: that in this digital day and age, information about money is more important than money!
- ComputerGuru 3mo ago
  > I pay about $0.075 USD per kWh Around here electricity companies quote prices like yours but that is supply only while transmission, taxes, and fees are again as much on top. Is that really all inclusive?
- pizza234 3mo ago
  Which are those use cases, considering that the hypothetical server, as described by the GP, is also extremely slow (4/6 hours for a response)?
cobbzilla 3mo ago
I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.
- sebmellen 3mo ago
  (Context: Parent comment was edited after I wrote this comment) Where in the world are you finding that much RAM in a racked server for $200/month?
  - walrus01 3mo ago
    I think he means electrical bill at his estimated wattage load of the server and his known kWh cost, not rented server/hosting cost.
    - sebmellen 3mo ago
      Aha, right. That makes a lot more sense.
  - cobbzilla 3mo ago
    At my house. I have 5Gbps fiber and could pay for 10 or 25 if I need it.
    - sebmellen 3mo ago
      Gotcha. But to be clear, you’re talking only about energy usage, correct?
btown 3mo ago
One aspect of this is cyberattack proliferation by way of "Hey boss, I saw this TikTok that says if you let me invest [a tiny piece of the neighborhood's profit|our militia's budget] into some RAM, I could get a fully autonomous cyber operation up and running that pays for itself via ransomware etc. within weeks. You like it, we upgrade to something that can work even faster. We don't need the hacker guy from Swordfish with fifty monitors, we just need my cousin who likes building gaming PCs." That's a world that I don't think we're ready for.
- 9x39 3mo ago
  A similar world is already here. Young men 14-?? already compromise and attempt to extort organizations daily, sometimes cluelessly from western nations, often not. It doesn’t have to be gangs when the home country doesn’t care / isn’t technologically or culturally developed. Already seeing AI-written payloads and frameworks in the wild. I think it’ll turn out that AI won’t build you a maintainable ERP but it can create C2 networks, exploit POCs or even 0-days potentially, and let kids make their own ransomware tooling. Then we’re dealing not with a handful of cybercrime tool makers but a generational problem.
tripzilch 3mo ago
> that justifies this this is hacker news sir we don't do things because they are easy ... or because they are justified ...
tripzilch 3mo ago
> extreme data privacy requirement so I have a a few decades worth of creative coding projects, many personal notes + writing, pictures, video clips, renders, projects, backups of the previous laptop's home folder that contains the laptop before it in the ~/dump folder and so on, spread out over several harddisks, laptops and storage media it's a mess that I've wanted to organize for years, but I can't seem to manage it but I might be able to sketch out a plan or task for a local multi-modal LLM to slowly chug through all the files (read-only), catalog and index them, copy to a clean external SSD, de-dupe and tidy everything up I mean, some of this data is already highly personal, and there might be stuff I forgot was there ... is it an "extreme data privacy requirement" to not want to share running an LLM for this task, and wanna do it locally?

by fooker 3mo ago

walrus01 3mo ago

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.

fooker 3mo ago
Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
- walrus01 3mo ago
  Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output. On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.
- sm-silversight 3mo ago
  >Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second? Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.
  - walrus01 3mo ago
    Very good description of it. It does seem like a bit of a rhetorical question to ask a forum that has a very high population of Linux and BSD users why they might desire to have the option to do something themselves rather than relying on an external packaged ready to go product.
- irishcoffee 3mo ago
  The whole mentality of thinking one knows better than another about what they need causes infinitely more problems than it solves.
light_hue_1 3mo ago
As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.
- walrus01 3mo ago
  Do you mean by trading dollars for the privacy you need as: a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place or b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place or c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds? Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.
  - jmalicki 3mo ago
    A) is very doable with e.g. Amazon Bedrock. They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights. What kind of privacy needs do you really have beyond that?
    - walrus01 3mo ago
      It is not my use case but given recent political developments in international relations caused by the executive branch of the US government, off the top of my head, I could think of a lot of European or Canadian firms for which that would not be an option. No matter what they might promise about European sovereignty. For a good 'ol patriotic US domestic company? Sure.
mdasen 3mo ago
That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents. There are parts of states like Grant County Washington that have cheap hydro power, but it's very rare for power to be that cheap in the US. Even if this applies to you, it won't apply to the vast majority of people on here who will have electric rates 2-4x higher. Average electric rates by region: New England 28.1 cents Mid Atlantic 25.1 cents East North Central 20.8 cents West North Central 14.8 cents South Atlantic 16.1 cents East South Central 15.5 cents Mountain 14.6 cents Pacific Contiguous 26.1 cents Pacific Noncontiguous 42.1 cents https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
- peterleiser 3mo ago
  Nashville Electric Service rates are 12.0 cents / kWh. Flat rate, no peak or off peak rates for residential.
- AgentMatt 3mo ago
  They are most likely not based in the US, but converting to USD to make comparison easier.
  - walrus01 3mo ago
    I am not in Quebec but Quebec hydro rate D for standard residential would be one example of around what I pay. https://www.hydroquebec.com/residential/customer-space/rates/rate-d.html https://www.hydroquebec.com/residential/customer-space/rates... Another example would be Manitoba hydro All figures in Canadian currency https://www.hydro.mb.ca/account/rates/residential/ https://www.hydro.mb.ca/account/rates/residential/
  - brailsafe 3mo ago
    Specifying USD is indeed often a service usually offered by people born elsewhere for people born elsewhere. Americans seem rarely know about these mysterious places, where bills can come in all sorts of funny sizes and colours. (kind of joking)
ljlolel 3mo ago
can send safely context if there’s confidential computing ala my site https://trustedrouter.com/ https://trustedrouter.com/
- skinfaxi 3mo ago
  How do you prove you are running exclusively on Nitro enclave instances or GCP confidential spaces?
  - Cider9986 3mo ago
    Is your local compute airgapped?
    - well_ackshually 3mo ago
      I trust my local compute quite a bit more than a random project that has "trusted" in its name.
    - trollbridge 3mo ago
      My local compute is used by me, and I'm accountable to myself whether or not is secure. So to a certain degree, I trust myself and also know what limitations / potential vulnerabilities it might have.
  - whs 3mo ago
    Seems clear from their website? 1. Their API server provide an attestation JWT. This JWT is signed by Google's private key. 2. The attestation has details on the running container. I suppose the container host is a Google-provided distro and Google's signer will verify that the OS is theirs and up-to-date. 3. They could've proxy the attestation. To prove this is not the case, the field eat_nonce include the TLS certificate fingerprint, which should match the API server you're connecting to. I suppose you will need to pull their container and verify from the source that the container itself generate the private key, it never leaves the container, and the container has no way to run arbitrary code such as SSH or vulnerabilities.
tiahura 3mo ago
There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. Are there? At the highest levels of defense and law, AWS and Azure are used. Having tried selling some of these entities on doing things in-house, there seems to be little interest.
- walrus01 3mo ago
  > Are there? At the highest levels of defense and law, AWS and Azure are used. This is certainly true if the user is an American company. You could look at the European initiatives to run this stuff on hardware they own in facilities they own and control within the borders of Europe for a counter-example. Such as: https://www.google.com/search?client=firefox-b-d&q=schwarz+stackit+germany https://www.google.com/search?client=firefox-b-d&q=schwarz+s... https://www.dutchnews.nl/2026/04/government-turns-to-german-cloud-provider-to-cut-us-reliance/ https://www.dutchnews.nl/2026/04/government-turns-to-german-...
  - jmalicki 3mo ago
    Yeah, true European cloud providers for these kinds of things seem to be behind, and a lot of the ones offering data compliance at the level of AWS are small enough that it's a bit harder to trust they'll be around and will keep their promises. Hopefully that changes!
doctorpangloss 3mo ago
It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.
- trollbridge 3mo ago
  Yeah. At 5 tok/second, you're talking about around $195 worth of output tokens per month. There is no way I can run a usable K3 model for $195 a month of capex, opex, or any-kind-of-ex. Qwen 3.6 is another matter. Paying provider rates for the amount I run locally would put me in the thousands of dollars. So that's very practical to buy a Macbook instead, plus an RTX card, and so on.
polycancel 3mo ago
>and people will compromise speed for data sovereignty People should always compromise speed for data sovereignty! Who said: that in this digital day and age, information about money is more important than money!
ComputerGuru 3mo ago
> I pay about $0.075 USD per kWh Around here electricity companies quote prices like yours but that is supply only while transmission, taxes, and fees are again as much on top. Is that really all inclusive?
pizza234 3mo ago
Which are those use cases, considering that the hypothetical server, as described by the GP, is also extremely slow (4/6 hours for a response)?

cobbzilla 3mo ago

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.

sebmellen 3mo ago
(Context: Parent comment was edited after I wrote this comment) Where in the world are you finding that much RAM in a racked server for $200/month?
- walrus01 3mo ago
  I think he means electrical bill at his estimated wattage load of the server and his known kWh cost, not rented server/hosting cost.
  - sebmellen 3mo ago
    Aha, right. That makes a lot more sense.
- cobbzilla 3mo ago
  At my house. I have 5Gbps fiber and could pay for 10 or 25 if I need it.
  - sebmellen 3mo ago
    Gotcha. But to be clear, you’re talking only about energy usage, correct?

btown 3mo ago

One aspect of this is cyberattack proliferation by way of "Hey boss, I saw this TikTok that says if you let me invest [a tiny piece of the neighborhood's profit|our militia's budget] into some RAM, I could get a fully autonomous cyber operation up and running that pays for itself via ransomware etc. within weeks. You like it, we upgrade to something that can work even faster. We don't need the hacker guy from Swordfish with fifty monitors, we just need my cousin who likes building gaming PCs." That's a world that I don't think we're ready for.

9x39 3mo ago
A similar world is already here. Young men 14-?? already compromise and attempt to extort organizations daily, sometimes cluelessly from western nations, often not. It doesn’t have to be gangs when the home country doesn’t care / isn’t technologically or culturally developed. Already seeing AI-written payloads and frameworks in the wild. I think it’ll turn out that AI won’t build you a maintainable ERP but it can create C2 networks, exploit POCs or even 0-days potentially, and let kids make their own ransomware tooling. Then we’re dealing not with a handful of cybercrime tool makers but a generational problem.

tripzilch 3mo ago

> that justifies this this is hacker news sir we don't do things because they are easy ... or because they are justified ...

tripzilch 3mo ago

> extreme data privacy requirement so I have a a few decades worth of creative coding projects, many personal notes + writing, pictures, video clips, renders, projects, backups of the previous laptop's home folder that contains the laptop before it in the ~/dump folder and so on, spread out over several harddisks, laptops and storage media it's a mess that I've wanted to organize for years, but I can't seem to manage it but I might be able to sketch out a plan or task for a local multi-modal LLM to slowly chug through all the files (read-only), catalog and index them, copy to a clean external SSD, de-dupe and tidy everything up I mean, some of this data is already highly personal, and there might be stuff I forgot was there ... is it an "extreme data privacy requirement" to not want to share running an LLM for this task, and wanna do it locally?