7 ms·
Some practical notes from digging around in their documentation: In order to get access to this, you need to be on their tier 5 level, which requires $1,000 tot
by OkGoDoIt 2y ago
Some practical notes from digging around in their documentation:
In order to get access to this, you need to be on their tier 5 level, which requires $1,000 total paid and 30+ days since first successful payment.
Pricing is $15.00 / 1M input tokens and $60.00 / 1M output tokens. Context window is 128k token, max output is 32,768 tokens.
There is also a mini version with double the maximum output tokens (65,536 tokens), priced at $3.00 / 1M input tokens and $12.00 / 1M output tokens.
The specialized coding version they mentioned in the blog post does not appear to be available for use.
It’s not clear if the hidden chain of thought reasoning is billed as paid output tokens. Has anyone seen any clarification about that? If you are paying for all of those tokens it could add up quickly. If you expand the chain of thought examples on the blog post they are extremely verbose.
https://platform.openai.com/docs/models/o1 https://platform.openai.com/docs/models/o1
https://openai.com/api/pricing/ https://openai.com/api/pricing/
https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-five https://platform.openai.com/docs/guides/rate-limits/usage-ti...
- activatedgeek 2y agoReasoning tokens are indeed billed as output tokens. > While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens. From here: https://platform.openai.com/docs/guides/reasoning https://platform.openai.com/docs/guides/reasoning
- baq 2y agoThis is concerning - how do you know you aren’t being fleeced out of your money here…? You’ll get your results, but did you really use that much?
- rsanek 2y agoobfuscated billing has long been a staple of all great cloud products. AWS innovated in the space and now many have followed in their footsteps
- lolinder 2y agoAlso, now we're paying for output tokens that aren't even output, with no good explanation for why these tokens should be hidden from the person who paid for them.
- HeatrayEnjoyer 2y agoIf you read the link they have a section specifically explaining why it is hidden.
- lolinder 2y agoI read it. It's a bad explanation. The only bit about it that feels at all truthful is this bit, which is glossed over but likely the only real factor in the decision: > after weighing multiple factors including ... competitive advantage ... we have decided not to show the raw chains of thought to users.
- HeatrayEnjoyer 2y agoBad, in your opinion.
- infogulch 2y agoGood catch. That indicates that chains of thought are a straightforward approach to make LLMs better at reasoning if you could copy it just by seeing the steps.
- RobertDeNiro 2y agoAlso seems very impractical to embed this into a deployed product. How can you possibly hope to control and estimate costs? I guess this is strictly meant for R&D purposes.
- sebzim4500 2y agoYou can specify the max length of the response, which presumably includes the hidden tokens. I don't see why this is qualitatively different from a cost perspective than using CoT prompting on existing models.
- dartos 2y agoYou can’t verify that you’re paying what you should be if you can’t see the hidden tokens.
- sebzim4500 2y agoWith the conventional models you don't get the activations or the logits even though those would be useful. Ultimately if the output of the model is not worth what you end up paying for it then great, I don't see why it really matters to you whether OpenAI is lying about token counts or not.
- dartos 2y agoAs a single user, it doesn’t really, but as a SaaS operator I want tractable, hopefully predictable pricing. I wouldn’t just implicitly trust a vendor when they say “yeah we’re just going to charge you for what we feel like when we feel like. You can trust us.”
- BoorishBears 2y agoFor one, you don't get to see any output at all if you run out of tokens during thinking. If you set a limit, once it's hit you just get a failed request with no introspection on where and why CoT went off the rails
- Emiledel 2y agoIn the UI the reasoning is visible. The API can probably return it too, just check the code
- famouswaffles 2y agoWhat's shown in the UI is a summary of the reasoning
- AlphaAndOmega0 2y agoOAI doesn't show the actual COT, on the grounds that it's potentially unsafe output and also to prevent competitors training on it. You only see a sanitized summary.
- jstummbillig 2y agoI think it's fantastic that now, for very little money, everyone gets to share a narrow but stressful subset of what it feels like to employ other people. Really, I recommend reading this part of the thread while thinking about the analogy. It's great.
- ta8645 2y agoYour idea is really a brilliant insight. Revealing.
- gsbcbdjfncnjd 2y agoAny respectable employer/employee relationship transacts on results rather than time anyway. Not sure the analogy is very applicable in that light.
- konschubert 2y agoIt is!
- adwn 2y ago> Any respectable employer/employee relationship transacts on results rather than time anyway. No. This may be common in freelance contracts, but is almost never the case in employment contracts, which specify a time-based compensation (usually either per hour or per month).
- ethbr1 2y agoI believe parent's point was that if ones management is clueless as to how to measure output and compensation/continued employment is unlinked from same... one is probably working for a bad company.
- gsbcbdjfncnjd 2y agoYea, I said ‘respectable’.
- 2y ago
- creatonez 2y agoNo access to reasoning output seems totally bonkers. All of the real cost is in inference, assembling an HTTP request to deliver that result seems trivial?
- amrrs 2y agoThe CoT is billed as output tokens. Mentioned in the docs where it talks about reasoning
- vdfs 2y agoWe just receivied this email: Hi there, I’m x, PM for the OpenAI API. I’m pleased to share with you our new series of models, OpenAI o1. We’ve developed these models to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math. As a trusted developer on usage tier 5, you’re invited to get started with the o1 beta today. Read the docs You have access to two models: Our larger model, o1-preview, which has strong reasoning capabilities and broad world knowledge. Our smaller model, o1-mini, which is 80% cheaper than o1-preview. Try both models! You may find one better than the other for your specific use case. Both currently have a rate limit of 20 RPM during the beta. But keep in mind o1-mini is faster, cheaper, and competitive with o1-preview at coding tasks (you can see how it performs here). We’ve also written up more about these models in our blog post. I’m curious to hear what you think. If you’re on X, I’d love to see what you build—just reply to our post. Best, OpenAI API
- sashank_1509 2y agoI have access to this and there is no way I spend more than 50$ on OpenAI api. I have ChatGPT + since day q though (240$ probably in total)
- rpmisms 2y agoYou missed your raise key on "day q"
- thelastparadise 2y agoRaise it up just one
- Buttons840 2y agoI am a Plus user and pay $20 per month. I have access to the o1 models.
- liamwire 2y agoUnless this is specifically relating to API access, I don’t think it’s correct. I’ve been paying for ChatGPT via the App Store IAP for around a year or less, and I’ve already got both o1-preview and o1-mini available in-app.
- OkGoDoIt 2y agoYes, I was referring to API access specifically. Nothing in the blog post or the documentation mentions access to these new models on ChatGPT, and even as a paid user I’m not seeing them on there (Edit: I am seeing it now in the app). But looks like a bunch of other people in this discussion do have it on ChatGPT, so that’s exciting to hear.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- arnaudsm 2y agoSome of the queries run for multiple minutes. 40 tokens/sec is too slow for CoT. I hope OpenAI is investing in low-latency like Groq's tech that can reach 1k tokens/sec.
- p-e-w 2y agoIt's slow and expensive if you compare it with other LLMs. It's lightning fast and dirt cheap if you compare it to consulting with a human expert, which it appears to be competitive with.
- barrell 2y agoI would say consulting with a human. Any expert who has a conversation with chatGPT about their field will verify that it is very far from expert
- p-e-w 2y agoAccording to the data provided by OpenAI, that isn't true anymore. And I trust data more than anecdotal claims made by people whose job is being threatened by systems like these.
- adonese 2y ago>According to the data provided by OpenAI, that isn't true anymore OpenAI main job is to sell that their models are better than human. I still remember when they're marketing their gpt-2 weights as too dangerous to release.
- mptest 2y agoI remember that too, it's when I started following the space (shout out computerphile/robert miles) and iirc the reason they gave was not "it's too dangerous cause it's so badass" they basically were correct in that it can produce sufficiently "human" output as to break typical bot detectors on social media which is a legitimate problem - whether the repercussions of that failure to detect botting is meaningful enough to be considered "dangerous" is up to the reader to decide also worth noting I don't agree with the comment you're replying to - but did want to add context to the situation of gpt-2
- anigbrowl 2y agoyou need to be on their tier 5 level, which requires $1,000 total paid and [...] Good opening for OpenAI's competitors to run a 'we're not snobs' promotion.
- infecto 2y agoHow so? I think most of the competition does this. Early partners/heavy users get access first which 1) hopefully provides feedback on the product and 2) provides a mechanism to stagger the release.
- jonahx 2y agoI am an ordinary plus user (since it was released more or less) and have access.
- mistersquid 2y ago> Some practical notes from digging around in their documentation: In order to get access to this, you need to be on their tier 5 level, which requires $1,000 total paid and 30+ days since first successful payment. Tier 5 level required for _API access_. ChatGPT Plus users, for example, also have access to the o1 models.
- attentive 2y agoSo, basically, it's chain of thought as a service? Not a model, per se, but a service that chains multiple model requests behind the scene?
- KeplerBoy 2y agoWho knows? Certainly not the public. It might be a finetuned model that works better in such a setting.
- OkGoDoIt 2y agoThe linked blog posts explains that it is fine-tuned on some reinforcement learning process. It doesn’t go into details but they do claim it’s not just the base model with chain of thought, there’s some fine-tuning going on.
- kinj28 2y agoA bit out of context. Am curious if at some point length of context window stops playing any material difference in the output and it just stops making any economical sense as law of marginal diminishing utility kicks in.
- kordlessagain 2y agoI'm a bit late to the show, but it would seem the API calls for these new models don't support system messages (where role is system) or the tool list for function calls.