3 ms·

"Inference" is just software running. It has always cost money to run software, it's just that it is generally too cheap to matter. If a client makes a regular

by chr15m 2mo ago

"Inference" is just software running. It has always cost money to run software, it's just that it is generally too cheap to matter. If a client makes a regular API call to your server, you pay for that compute, probably in the form of a flat hosting fee. If too many calls come in and workload goes up, you pay for a more expensive hosting tier to handle it (or do dynamic scaling which is per-unit of compute). Right now the "hosting" cost for inference is per-unit because it's new and expensive, but that won't last. There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app. The bigger problem for SaaS is that the floor has risen - people can build their own solutions for things that they used to buy SaaS for. So the industry needs to level up and solve harder problems.

euazOn 2mo ago
Exactly my point in another comment. Just to illustrate this further: a rough ballpark of how the cost of intelligence fell since 2022 could be about 1000x, and continues to fall. Unfortunately, it’s really hard to measure. It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
margalabargala 2mo ago
> There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app. I don't know about "very fast" or "soon" unless you're speaking in geological terms. SOTA models like Kimi 3 require thousands of GB of RAM/VRAM to run at speeds that are real-time useful. Manufacturing the memory necessary for that quantity to be available at app-hosting prices will take decades. Software efficiency solutions might drop needed memory by an order of magnitude in that time...but a tenth of an enormous amount is still pretty darn big so won't get us there "soon".
- dalenw 2mo ago
  Speaking of soon, tangentially related this just got announced: https://standardcode.ai/ https://standardcode.ai/ They claim to support 24/7 agent coding with no VC subsidizations or money lost on a subscription, instead relying on optimizations on agent selection. I think it'll shift sooner rather than later.
  - yencabulator 2mo ago
    They also don't seen to make any claims on how fast the LLM will respond? Take Qwen3.6 or such, figure out how many users you can batch-share one server between. server_cost_per_month / users_per_server = monthly_cost. Apparently, H200s are $1.57/h with commitments or <$2/h with spot pricing, so server_cost_per_month=1200 maybe (not my area). batch_throughput=3400tps apparently, assume 15% utilization of the actual LLM (building, testing, human typing, idle, etc; remember, no concurrency offered, "1 line") at SLO 100tps you could pack about users_per_server=227 for $5.28/user/month cost. Easy to build rate limiting to throttle users to a fixed max speed. At that point providing the actual VM where the code runs is a comparable cost. Actual measurements may vary, but batching is the gimmick. (Whether Qwen3.6 is good enough to pay for is kinda beside the point; they're specifically avoiding saying what model it is, so don't expect you're getting full-speed SOTA 24/7. You don't know whether they're selling you Qwen3.6 or what. They're also apparently forcing you into their custom harness, system prompts, virtual machines... very little here says anything about whether it works well or not. Also no specs about the VM they put you on. The fact that they say "VPS" scares me.)
j16sdiz 2mo ago
> That will change very fast and soon paying for inference will likely resemble paying for hosting your app. The article have talked about this: > You cannot, however, build a business model today on a cost breakthrough that has not happened yet. Even if costs do fall, Jevons paradox[5] tends to kick in: historically, when a resource gets cheaper, we find more uses for it rather than consuming less. Cheaper inference does not get banked as margin improvement; it gets consumed by deeper integration, more calls per interaction, more autonomous agents running in the background. The cost per call drops but the calls per user multiply.

by chr15m 2mo ago

euazOn 2mo ago

Exactly my point in another comment. Just to illustrate this further: a rough ballpark of how the cost of intelligence fell since 2022 could be about 1000x, and continues to fall. Unfortunately, it’s really hard to measure. It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.

margalabargala 2mo ago

> There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app. I don't know about "very fast" or "soon" unless you're speaking in geological terms. SOTA models like Kimi 3 require thousands of GB of RAM/VRAM to run at speeds that are real-time useful. Manufacturing the memory necessary for that quantity to be available at app-hosting prices will take decades. Software efficiency solutions might drop needed memory by an order of magnitude in that time...but a tenth of an enormous amount is still pretty darn big so won't get us there "soon".

dalenw 2mo ago
Speaking of soon, tangentially related this just got announced: https://standardcode.ai/ https://standardcode.ai/ They claim to support 24/7 agent coding with no VC subsidizations or money lost on a subscription, instead relying on optimizations on agent selection. I think it'll shift sooner rather than later.
- yencabulator 2mo ago
  They also don't seen to make any claims on how fast the LLM will respond? Take Qwen3.6 or such, figure out how many users you can batch-share one server between. server_cost_per_month / users_per_server = monthly_cost. Apparently, H200s are $1.57/h with commitments or <$2/h with spot pricing, so server_cost_per_month=1200 maybe (not my area). batch_throughput=3400tps apparently, assume 15% utilization of the actual LLM (building, testing, human typing, idle, etc; remember, no concurrency offered, "1 line") at SLO 100tps you could pack about users_per_server=227 for $5.28/user/month cost. Easy to build rate limiting to throttle users to a fixed max speed. At that point providing the actual VM where the code runs is a comparable cost. Actual measurements may vary, but batching is the gimmick. (Whether Qwen3.6 is good enough to pay for is kinda beside the point; they're specifically avoiding saying what model it is, so don't expect you're getting full-speed SOTA 24/7. You don't know whether they're selling you Qwen3.6 or what. They're also apparently forcing you into their custom harness, system prompts, virtual machines... very little here says anything about whether it works well or not. Also no specs about the VM they put you on. The fact that they say "VPS" scares me.)

j16sdiz 2mo ago

> That will change very fast and soon paying for inference will likely resemble paying for hosting your app. The article have talked about this: > You cannot, however, build a business model today on a cost breakthrough that has not happened yet. Even if costs do fall, Jevons paradox[5] tends to kick in: historically, when a resource gets cheaper, we find more uses for it rather than consuming less. Cheaper inference does not get banked as margin improvement; it gets consumed by deeper integration, more calls per interaction, more autonomous agents running in the background. The cost per call drops but the calls per user multiply.