4 ms·

A useful feature would be slow-mode which gets low cost compute on spot pricing. I’ll often kick off a process at the end of my day, or over lunch. I don’t nee

A useful feature would be slow-mode which gets low cost compute on spot pricing. I’ll often kick off a process at the end of my day, or over lunch. I don’t need it to run immediately. I’d be fine if it just ran on their next otherwise-idle gpu at much lower cost that the standard offering.

guerrilla 8mo ago
> I’ll often kick off a process at the end of my day, or over lunch. I don’t need it to run immediately. I’d be fine if it just ran on their next otherwise-idle gpu at much lower cost that the standard offering. If it's not time sensitive, why not just run it at on CPU/RAM rather than GPU.
- bethekidyouwant 8mo ago
  Run what exactly?
  - all2 8mo ago
    I'm assuming GP means 'run inference locally on GPU or RAM'. You can run really big LLMs on local infra, they just do a fraction of a token per second, so it might take all night to get a paragraph or two of text. Mix in things like thinking and tool calls, and it will take a long, long time to get anything useful out of it.
    - guerrilla 8mo ago
      Yes, this is what I meant. People are running huge models at home now, I assumed people could do it on premises or in a data center if you're a business, presumably faster... but yeah it definitely depends on what time scales we're talking.
      - HumanOstrich 8mo ago
        Huge models? First you have to spend $5k-$10k or more on hardware. Maybe $3k for something extremely slow (<1 tok/sec) that is disk-bound. So that's not a great deal over batch API pricing for a long, long time. Also you still wouldn't be able to run "huge" models at a decent quantization and token speed. Kimi K2.5 (1T params) with a very aggressive quantization level might run on one Mac Studio with 512GB RAM at a few tokens per second. To run Kimi K2.5 at an acceptable quantization and speed, you'd need to spend $15k+ on 2 Mac Studios with 512GB RAM and cluster them. Then you'll maybe get 10-15 tok/sec.
      - copperx 8mo ago
        I'd love to know what kind of hardware would it take to do inference at the speed provided by the frontier model providers (assuming their models were available for local use). 10k worth of hardware? 50k? 100k? Assuming a single user.
    - hxtk 8mo ago
      I’ve been experimenting with this today. I still don’t think AI is a very good use of my programming time… but it’s a pretty good use of my non-programming time. I ran OpenCode with some 30B local models today and it got some useful stuff done while I was doing my budget, folding laundry, etc. It’s less likely to “one shot” apples to apples compared to the big cloud models; Gemini 3 Pro can one shot reasonably complex coding problems through the chat interface. But through the agent interface where it can run tests, linters, etc. it does a pretty good job for the size of task I find reasonable to outsource to AI. This is with a high end but not specifically AI-focused desktop that I mostly built with VMs, code compilation tasks, and gaming in mind some three years ago.
- weird-eye-issue 8mo ago
  Yeah just run a LLM with over 100 billion parameters on a CPU.
  - kristjansson 8mo ago
    200 GB is an unfathomable amount of main memory for a CPU (with apologies for snark,) give gpt-oss-120b a try. It’s not fast at all, but it can generate on CPU.
    - awestroke 8mo ago
      But it's incredibly incapable compared to SOTA models. OP wants high quality output but doesn't need it fast. Your suggestion would mean slow AND low quality output.
      - kristjansson 8mo ago
        Set your parameters to make that point then. “Yeah just run a 1T+ model on CPU”
- gruez 8mo ago
  Does that even work out to be cheaper, once you factor in how much extra power you'd need?
  - HumanOstrich 8mo ago
    How much extra power do you think you would need to run an LLM on a CPU (that will fit in RAM and be useful still)? I have a beefy CPU and if I ran it 24/7 for a month it would only cost about $30 in electricity.
stavros 8mo ago
OpenAI offers that, or at least used to. You can batch all your inference and get much lower prices.
- airspresso 8mo ago
  Still do. Great for workloads where it's okay to bundle a bunch of requests and wait some hours (up to 24h, usually done faster) for all of them to complete.
spondyl 8mo ago
https://platform.claude.com/docs/en/build-with-claude/batch-processing https://platform.claude.com/docs/en/build-with-claude/batch-... > The Batches API offers significant cost savings. All usage is charged at 50% of the standard API prices.
- jaytaylor 8mo ago
  Can this work for Claude? I think it might be raw API only.
  - spondyl 8mo ago
    I'm not sure I understand the question? Are you perhaps asking if messages can be batched via Claude Code and/or the Claude web UI?
    - jaytaylor 8mo ago
      Yes, Claude code.
      - HumanOstrich 8mo ago
        No
gardnr 8mo ago
The discount MAX plans are already on slow-mode.
mrklol 8mo ago
Yep same, I often think why this isn’t a thing yet. Running some tasks in the night at e.g. 50% of the costs - there’s the batch api but that is not integrated in e.g. claude code

guerrilla 8mo ago

> I’ll often kick off a process at the end of my day, or over lunch. I don’t need it to run immediately. I’d be fine if it just ran on their next otherwise-idle gpu at much lower cost that the standard offering. If it's not time sensitive, why not just run it at on CPU/RAM rather than GPU.

bethekidyouwant 8mo ago
Run what exactly?
- all2 8mo ago
  I'm assuming GP means 'run inference locally on GPU or RAM'. You can run really big LLMs on local infra, they just do a fraction of a token per second, so it might take all night to get a paragraph or two of text. Mix in things like thinking and tool calls, and it will take a long, long time to get anything useful out of it.
  - guerrilla 8mo ago
    Yes, this is what I meant. People are running huge models at home now, I assumed people could do it on premises or in a data center if you're a business, presumably faster... but yeah it definitely depends on what time scales we're talking.
    - HumanOstrich 8mo ago
      Huge models? First you have to spend $5k-$10k or more on hardware. Maybe $3k for something extremely slow (<1 tok/sec) that is disk-bound. So that's not a great deal over batch API pricing for a long, long time. Also you still wouldn't be able to run "huge" models at a decent quantization and token speed. Kimi K2.5 (1T params) with a very aggressive quantization level might run on one Mac Studio with 512GB RAM at a few tokens per second. To run Kimi K2.5 at an acceptable quantization and speed, you'd need to spend $15k+ on 2 Mac Studios with 512GB RAM and cluster them. Then you'll maybe get 10-15 tok/sec.
    - copperx 8mo ago
      I'd love to know what kind of hardware would it take to do inference at the speed provided by the frontier model providers (assuming their models were available for local use). 10k worth of hardware? 50k? 100k? Assuming a single user.
  - hxtk 8mo ago
    I’ve been experimenting with this today. I still don’t think AI is a very good use of my programming time… but it’s a pretty good use of my non-programming time. I ran OpenCode with some 30B local models today and it got some useful stuff done while I was doing my budget, folding laundry, etc. It’s less likely to “one shot” apples to apples compared to the big cloud models; Gemini 3 Pro can one shot reasonably complex coding problems through the chat interface. But through the agent interface where it can run tests, linters, etc. it does a pretty good job for the size of task I find reasonable to outsource to AI. This is with a high end but not specifically AI-focused desktop that I mostly built with VMs, code compilation tasks, and gaming in mind some three years ago.
weird-eye-issue 8mo ago
Yeah just run a LLM with over 100 billion parameters on a CPU.
- kristjansson 8mo ago
  200 GB is an unfathomable amount of main memory for a CPU (with apologies for snark,) give gpt-oss-120b a try. It’s not fast at all, but it can generate on CPU.
  - awestroke 8mo ago
    But it's incredibly incapable compared to SOTA models. OP wants high quality output but doesn't need it fast. Your suggestion would mean slow AND low quality output.
    - kristjansson 8mo ago
      Set your parameters to make that point then. “Yeah just run a 1T+ model on CPU”
gruez 8mo ago
Does that even work out to be cheaper, once you factor in how much extra power you'd need?
- HumanOstrich 8mo ago
  How much extra power do you think you would need to run an LLM on a CPU (that will fit in RAM and be useful still)? I have a beefy CPU and if I ran it 24/7 for a month it would only cost about $30 in electricity.

stavros 8mo ago

OpenAI offers that, or at least used to. You can batch all your inference and get much lower prices.

airspresso 8mo ago
Still do. Great for workloads where it's okay to bundle a bunch of requests and wait some hours (up to 24h, usually done faster) for all of them to complete.

spondyl 8mo ago

https://platform.claude.com/docs/en/build-with-claude/batch-processing https://platform.claude.com/docs/en/build-with-claude/batch-... > The Batches API offers significant cost savings. All usage is charged at 50% of the standard API prices.

jaytaylor 8mo ago
Can this work for Claude? I think it might be raw API only.
- spondyl 8mo ago
  I'm not sure I understand the question? Are you perhaps asking if messages can be batched via Claude Code and/or the Claude web UI?
  - jaytaylor 8mo ago
    Yes, Claude code.
    - HumanOstrich 8mo ago
      No

gardnr 8mo ago

The discount MAX plans are already on slow-mode.

mrklol 8mo ago

Yep same, I often think why this isn’t a thing yet. Running some tasks in the night at e.g. 50% of the costs - there’s the batch api but that is not integrated in e.g. claude code