5 ms·
50% discount on OpenAI pricing if you submit a batch and give them up to 24h
- Marlinski 3y agoWish this was done over Bundle Protocol, would love to code that for them!
- ravetcofx 3y agoWe've gone full circle to an era of Mainframe timeshare scheduling tasks.
- VBprogrammer 3y agoGive it 10 years and your mid-price android phone will be running models that would make ChatGPT look like baby talk.
- ravetcofx 3y agoMaybe even talking birthday cards with LLM ASICs in a blob
- 999900000999 3y agoHow does this work, do I submit a rest call and get an IOU response to look up the results later?
- deleted 3y ago[deleted]
- tedsanders 3y agoCopied from OpenAI's Jeff Harris on Twitter: 1) Upload your batch file curl https://api.openai.com/v1/files https://api.openai.com/v1/files -H "Authorization: Bearer $OPENAI_API_KEY" -F purpose="batch" -F file="@batch_example.jsonl" 2) Create the batch job curl --request POST --url https://api.openai.com/v1/batches https://api.openai.com/v1/batches --header "Authorization: Bearer $OPENAI_API_KEY" --header 'Content-Type: application/json' --data '{ "input_file_id": "file-DYJImtYAQ2y3j25b5F27Eefp", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' 3) Check the status of your job curl https://api.openai.com/v1/batches/batch_f5Hh3MXaM0NNuTGOmutlSEmq https://api.openai.com/v1/batches/batch_f5Hh3MXaM0NNuTGOmutl... -H "Authorization: Bearer $OPENAI_API_KEY" 4) Download the completed file! curl https://api.openai.com/v1/files/file-Og2t4LcJOCqOFq6rmp22Uzke/content https://api.openai.com/v1/files/file-Og2t4LcJOCqOFq6rmp22Uzk... -H "Authorization: Bearer $OPENAI_API_KEY" > batch_output.jsonl
- 999900000999 3y agoCool, not really looking to do this just yet, but it's definitely on my radar the next time I have a project. Would love an option to look up all my open jobs. I imagine doing this in a ci/cd pipeline or GitHub action. But I don't want to have to store state.
- barbazoo 3y agoPresumably that would be a curl --request GET --url https://api.openai.com/v1/batches https://api.openai.com/v1/batches but that doesn't seem to exist yet: https://platform.openai.com/docs/api-reference/batch https://platform.openai.com/docs/api-reference/batch
- 999900000999 3y agoBut it looks like you can upload files and list them. I'd just upload a file with the name Timestamp_batchID, and get the list of files later.
- silvacarl 3y agothis is crazy cool, will try it out shortly. i need to batch anaylze 100K whisper transcriptions. 8-)
- mike_hearn 3y agoIt's a pity that this sort of feature is done with a custom ad-hoc HTTP protocol instead of using a message queuing protocol like AMQP or STOMP. Those are designed for this kind of thing and already have libraries available, they avoid the need for busywork polling and avoid the need to break work up into batches client side. You should just be able to submit messages containing work to a queue and then let the broker inform you when new responses arrive on the response queue if you're connected. If you're not connected they buffer up for a while until you reconnect and drain the queue. The advantage of that approach would be that latency could be minimized or tiered. Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is. I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON.
- dmw_ng 3y agoThere's a reason pretty much everything that does not require low latency replies avoids stateful networking - everything from RSS to video streaming prefers stateless polling designs because it is vastly easier for both parties to implement and scale. Meanwhile, I couldn't name a single system in widespread use built around a MQ paradigm in its public interface, except for actual MQ APIs, and many of those (e.g. from AWS) are still built on polling for the reasons just described
- jerf 3y agoBehind each request made to OpenAI is a staggering amount of GPU computation. If the price of the queue request is even a hundred thousandth of the overall price of a single request I'd be stunned. There is no message queue scaling issue here. Message queue scaling issues arise when you are blasting around a lot of messages, but each of them take minimal resources on an individual basis to service, so it's feasible for the queue itself to be the bottleneck. I wouldn't be surprised a single Raspberry Pi could handle the entire queuing load here, and if it couldn't it's not off by a very large factor, because the GPU resources behind what it would take to service a full RPi's queuing capacity would be staggeringly enormous, I think well beyond what OpenAI actually has.
- robtest123 3y ago[flagged]
- jasonjmcghee 3y agoOriginal / official tweet: https://twitter.com/openaidevs/status/1779922566091522492?s=46&t=ZjF9if4qtE9ppAnug9UOPQ https://twitter.com/openaidevs/status/1779922566091522492?s=... API Link: https://platform.openai.com/docs/api-reference/batch https://platform.openai.com/docs/api-reference/batch
- tony_cannistra 3y agoSuggests to me that they have a LOT of compute that they can't, or have decided not to, scale down. Interesting way for them to keep the infra up and cover costs.
- jstummbillig 3y agoIt mostly suggests large spikes in usage, no? Which does not seem very surprising.
- tony_cannistra 3y agoTotally. There are other explanations (like wanting to offer a discount), but my mind initally went to "they have a lot of provisioned compute that they pre-paid for, and they want higher-utilization when API request rates are low." Just hypothesizing.
- amluto 3y agoOr they don't have enough provisioned compute to cover peak usage, and they therefore want to reduce peak usage.
- blackoil 3y agoOr simply that traffic is uneven. Like the Spot instances of EC2.
- danielmarkbruce 3y agoI don't think that's right. There is an optimal batch size, and this allows them to just keep everything optimal or close to it for as much time as possible. I'd bet every single minute they miss chances to run things optimally because they need to hit some latency number, and this enables them to not have to make that trade off.
- tony_cannistra 3y agoYeah, that's another good explanation.
- victorbjorklund 3y agoCould def be useful for many use cases.
- hoseja 3y agoDon't spill the card stack!
- namaria 3y ago50 years of Moore's law improvements and we're back to batch processing. Nothing illustrates better to me the regression that using massive amounts of GPU to do inference represents as an approach to computation.
- beachy 3y ago> 50 years of Moore's law improvements and we're back to batch processing. It never went away. I did some consulting years ago for a large telco. They had an overnight billing batch process that was obviously critical to the company. It took about 22 hours to run. They were terrified it was going to start taking longer, if it ever reached 24 hours that would be the death of the company.
- stygiansonic 3y agoThis is probably using their excess capacity, but not necessarily that their GPUs are idle. For LLMs/large models the huge cost is memory ops to load each layer weights during the forward pass. This is why doing inference at batch size 1 is extremely wasteful: you pay all the mem ops cost and don’t use enough compute FLOPs to justify. You want a high enough batch size so that compute:mem ops is close to the ratio for that GPU. This is usually done by batching together multiple user requests. At times of low usage there is excess capacity because batch size is below this “optimal” ratio. So they can slot in these “relaxed SLA” requests for little marginal increase resource usage on their end. Basically have a queue of these requests that you use to “top off” your batch size when you can. Edit: also you may not be able to get optimal batch size depending on when the requests arrive, eg you don’t want to wait forever to fill up a batch. So again having a queue of outstanding/delayed requests to serve allows for smoothing things out and increasing compute utilization
- tony_cannistra 3y agoYeah this makes sense. I do wonder though how it changes the dynamics around provisioned capacity, if at all.
- mike_hearn 3y agoIt reduces the need. If they can get non-latency sensitive users onto this API then they only need to be provisioned to support their max interactive query load (ChatGPT) rather than peak API load, which can be arbitrary high (however fast the program generating the load can run). The lower pricing should move users across quite fast, and the higher efficiency will free up hardware and reduce the rate at which they need to grow it.
- tony_cannistra 3y agoThat's the way it seems to me as well. Curious too about the business implications. My guess is that they wanted to bite the bullet and commit to provisioned capacity but wanted to do so in a way that didn't require massive overprovisioning for API requests.
- nextworddev 3y agoProbably a response to provisioned capacity inference from aws gcp
- deleted 3y ago[deleted]
- alienicecream 3y ago[flagged]
- deleted 3y ago[deleted]