2 ms·
I think they're feeling the heat from growing too quickly so they want to incentivize people to spread the load more evenly.
by stavros 8mo ago
I think they're feeling the heat from growing too quickly so they want to incentivize people to spread the load more evenly.
- toomuchtodo 8mo agoVery much like electric utility time of day pricing, using economic incentives to shift demand to trough periods. Perhaps an opportunity for them to improve workload scheduling orchestration, like submitting a job to a distributed computing cluster queue, to smooth demand and maximize utilization.
- stavros 8mo agoEverything bursty will use economic incentives to smooth the load. I'm not sure how they'd do that with workload scheduling orchestration when you have latency-sensitive loads and there are e.g. twice as many requests at midday as at midnight.
- toomuchtodo 8mo agoYou decouple the workloads from human interaction (ie when you submit the job to the queue vs when it is scheduled to execute) so when they run is not a consideration, if possible. The economic incentives encourage solving this, and if it can’t be solved, it buckets customer cohort by willingness (or unwillingness) to pay for access during peak times.
- stavros 8mo agoSure, but if I ask the LLM a question, I'd like it to respond now, instead of tonight.
- toomuchtodo 8mo agoCertainly, interactive workloads aren’t realistic for time shifting, but agentic coding likely is. Package everything up and ship it as a job, getting a bundle back asynchronously.
- stavros 8mo agoI don't know, my agentic coding is pretty interactive. Maybe once the plan is done, sure. That would be interesting, though OpenAI already does this with batch workloads.