8 ms·

New tools for building agents

389 points by meetpateltech 2y ago

hodanli 2y ago
I wonder why they phased out Pydantic in structured output for the Responses API.
- sdcoffey 2y ago
  Hey there! This is Steve here from the OpenAI team–I worked on the Responses API. We have not removed this! It should still work just like before! Here's an example: https://github.com/openai/openai-python/blob/main/examples/responses/structured_outputs.py https://github.com/openai/openai-python/blob/main/examples/r...
  - hodanli 2y ago
    Hi Steve, I based my assumption on the examples in the documentation[^1]. It is great that we can still use that. 1. https://platform.openai.com/docs/guides/structured-outputs?api-mode=responses https://platform.openai.com/docs/guides/structured-outputs?a...
- lunarcave 2y ago
  (Shameless plug) I worked on something for anyone else wanting to get structured outputs from LLMs in a model agnostic way (Including Open AI models): https://github.com/inferablehq/l1m https://github.com/inferablehq/l1m
nowittyusername 2y ago
How does this compare to MCP? Anyone has any considerations on the matter?
nnurmanov 2y ago
Does anyone know if there are any difference if you typed the question with typos vs you did it correctly?
- davidbarker 2y ago
  In theory there shouldn't be — LLMs are pretty robust to typos and usually infer the intended meaning regardless.
swyx 2y ago
swyx here. we got some preview and time with the API/DX team to ask FAQs about all the new APIs. https://latent.space/p/openai-agents-platform https://latent.space/p/openai-agents-platform main fun part - since responses are stored for free by default now, how can we abuse the Responses API as a database :) other fun qtns that a HN crew might enjoy: - hparams for websearch - depth/breadth of search for making your own DIY Deep Research - now that OAI is offering RAG/reranking out of the box as part of the Responses API, when should you build your own RAG? (i basically think somebody needs to benchmark the RAG capabilities of the Files API now, because the community impression has not really updated from back when Assistants API was first launched) - whats the diff between Agents SDK and OAI Swarm? (basically types, tracing, pluggable LLMs) - will the `search-preview` and `computer-use-preview` finetunes be merged into GPT5?
- ggnore7452 2y ago
  appreciate the question on hparams for websearch! one of the main reasons i build these ai search tools from scratch is that i can fully control the depth and breadth (and also customize loader to whatever data/sites). and currently the web search isn't very transparent on what sites they do not have full text or just use snippets. having computer use + websearch is definitely something very powerful (openai's deep research essentially)
- mritchie712 2y ago
  for anyone that likes the Agents SDK, but doesn't want their framework attached to OpenAI, we're really liking PydanticAI[0]. 0 - https://ai.pydantic.dev/ https://ai.pydantic.dev/
  - fullstackwife 2y ago
    Openai SDK docs: > Notably, our SDK is compatible with any model providers that support the OpenAI Chat Completions API format. so you can use with everything, not only OpenAI?
    - DrBenCarson 2y ago
baxtr 2y ago
A bit off topic but the post comes handy: can we settle the debate what an agent really is? It seems like everyone has their own definition. Ok I’ll start: an agent is a computer program that utilized LLMs heutiger for decision making.
- codydkdc 2y ago
  an agent is software that does something on behalf of someone (aka software) I personally strongly prefer the term "bots" for what most of these frameworks call "agents"
  - handfuloflight 2y ago
    Stick to the agentic nomenclature if you want at least a magnitude increase in valuation.
- deleted 2y ago
  [deleted]
- 3stripe 2y ago
  First rule of writing definitions: use everyday English.
  - baxtr 2y ago
    True! Meant heuristic
- knowaveragejoe 2y ago
  I think Anthropic's definition makes the most sense. - Workflows are systems where LLMs and tools are orchestrated through predefined code paths. (imo this is what most people are referring to as "agents") - Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks. https://www.anthropic.com/engineering/building-effective-agents https://www.anthropic.com/engineering/building-effective-age...
zellyn 2y ago
Notably not mentioned: Model Context Protocol https://www.anthropic.com/news/model-context-protocol https://www.anthropic.com/news/model-context-protocol
- esafak 2y ago
  How do they compare?
  - cowpig 2y ago
    MCP is a protocol, and Anthropic has provided SDKs for implementing that protocol. In practice, I find the MCP protocol to be pretty great, but it leaves basically everything except the model parts out. I.e. MCP really only addresses how "agentic" systems interact with one another, nothing else. This SDK is trying to provide a bunch of code for implementing specific agent codebases. There are a bunch of open source ones already, so this is OpenAI throwing their hat in the ring. IMO this OpenAI release is kind of ecosystem-hostile in that they are directly competing with their users, in the same way that the GPT apps were.
    - esafak 2y ago
      Thank you. Which open source ones do you recommend?
      - cowpig 2y ago
        Here's a fairly comprehensive list: https://github.com/slavakurilyak/awesome-ai-agents https://github.com/slavakurilyak/awesome-ai-agents CrewAI is a popular VC-backed one, but two that I think are kind of interesting in the open source space are: https://github.com/i-am-bee/beeai-framework https://github.com/i-am-bee/beeai-framework https://github.com/lastmile-ai/mcp-agent https://github.com/lastmile-ai/mcp-agent ... However I think the vast majority of "AI Agent" use-cases in practice right now are actually just workflows, and imo dify is great for those: https://github.com/langgenius/dify https://github.com/langgenius/dify [edit] worth mentioning [langfuse](https://github.com/langfuse/langfuse https://github.com/langfuse/langfuse), which is more like a platform that addresses the observability/evals/prompt management piece of the puzzle as opposed to a full-on "agent framework". In practice I have not yet run into a case where I needed something like what OpenAI just released, nor crewAI etc (despite it feeling like those cases may be coming)
rvz 2y ago
They did not announce the price(s) in the presentation. Likely because they know it is going to be very expensive: Web Search [0] * $30 and $25 per 1K queries for GPT‑4o search and 4o-mini search. File search [1] * $2.50 per 1K queries and file storage at $0.10/GB/day * First 1GB is free. Computer use tool (computer-use-preview model) [2] * $3 per 1M input tokens and $12/1M output tokens. [0] https://platform.openai.com/docs/pricing#web-search https://platform.openai.com/docs/pricing#web-search [1] https://platform.openai.com/docs/pricing#built-in-tools https://platform.openai.com/docs/pricing#built-in-tools [2] https://platform.openai.com/docs/pricing#latest-models https://platform.openai.com/docs/pricing#latest-models
- yard2010 2y ago
  So they're basically pivoting from selling text by the ounce to selling web searches and cloud storage? I like it, it's a bold move. When the slow people at Google finally catch up it might be too late for Google?
  - KoolKat23 2y ago
    Google AI Studios "Grounding" basically web search is priced similarly. (Very expensive for either, although Google gives you your first 1500 queries free). It seems completely upside down, they always said traditional search was cheaper/less intensive, I guess a lot of tokens must go into the actual LLM searching and retrieving.
- rudedogg 2y ago
  Those prices, especially for web search, are absurd. I have a hard time seeing how this API is better than https://www.anthropic.com/news/model-context-protocol https://www.anthropic.com/news/model-context-protocol. It seems like the motivation was "how can we make more money", rather than "how can we be more useful for our users".
- _bramses 2y ago
  For anyone who's looking Brave Search is $3/1000 requests [0] I also wrote a script that searches the web and works pretty well (using the vercel ai sdk)[1] [0] - https://brave.com/search/api/ https://brave.com/search/api/ [1] - https://gist.github.com/bramses/41e90b27d156590154bcefd4119f2b19 https://gist.github.com/bramses/41e90b27d156590154bcefd4119f...
Areibman 2y ago
Nice to finally see one of the labs throwing weight behind a much needed simple abstraction. It's clear they learned from the incumbents (langchain et al)-- don't sell complexity. Also very nice of them to include extensible tracing. The AgentOps integration is a nice touch to getting behind the scenes to understand how handoffs and tool calls are triggered
- swyx 2y ago
  why agentops specifically? doesnt the oai first party one also do it?
  - Areibman 2y ago
    The OpenAI dash is great but is clearly missing a lot of features (i.e. data export, alerts, non-oai model compatibility). Believe it or not, but they don't even report response API costs on spans
    - swyx 2y ago
      oh lol i wasnt looking at usernames. it's you! spiderman pointing
- esafak 2y ago
  Extensible how?
- deleted 2y ago
  [deleted]
serjester 2y ago
This is one of the few agent abstractions I've seen that actually seems intuitive. Props to the OpenAI team, seems like it'll kill a lot of bad startups.
- sdcoffey 2y ago
  Steve here from the OpenAI team–this means a lot! We really hope you enjoy building on it
ilaksh 2y ago
The Agents SDK they linked to comes up 404. BTW I have something somewhat similar to some of this like Responses and File Search in MindRoot by using the task API: https://github.com/runvnc/mindroot/blob/main/api.md https://github.com/runvnc/mindroot/blob/main/api.md Which could be combined with the query_kb tool from the mr_kb plugin (in my mr_kb repo) which is actually probably better than File Search because it allows searching multiple KBs. Anyway, if anyone wants to help with my program, create a plugin on PR, or anything, feel free to connect on GitHub, email or Discord/Telegram (runvnc).
- yablak 2y ago
  Loads fine for me. Maybe because I'm logged in?
  - IncreasePosts 2y ago
    That should be a 403 then. Tsk tsk open ai
    - 29ebJCyy 2y ago
      Technically it should be a 401. Tsk tsk IncreasePosts.
      - __float 2y ago
        It's common (see: S3, private GitHub repos) to return 404 instead of unauthorized to avoid even leaking existence of a resource at URL.
- edwinarbus 2y ago
  sorry, fixed link: https://openai.github.io/openai-agents-python/ https://openai.github.io/openai-agents-python/
anorak27 2y ago
I have built myself a much simpler and powerful version of the responses API and it works with all LLM providers. https://github.com/Anilturaga/aiide https://github.com/Anilturaga/aiide
- grvdrm 2y ago
  Thank you for your detailed Readme. A relief / joy to read compared to many other libraries/etc. that provide one basic (if that) example and otherwise leave to your own trial/error.
- bsenftner 2y ago
  For so few Github stars, I'm surprised that this is the 4th time I'm reading about your aiide project in 2 days. It looks good, very good BTW.
nextworddev 2y ago
This may be bad for Langflow, Langsmith, etc
mentalgear 2y ago
Well, I'll just wait 2-3 days until a (better) open-source alternative is released. :D
jumploops 2y ago
> “we plan to formally announce the deprecation of the Assistants API with a target sunset date in mid-2026.” The new Responses API is a step in the right direction, especially with the built-in “handoff” functionality. For agentic use cases, the new API still feels a bit limited, as there’s a lack of formal “guardrails”/state machine logic built in. > “Our goal is to give developers a seamless platform experience for building agents” It will be interesting to see how they move towards this platform, my guess is that we’ll see a graph-based control flow in the coming months. Now there are countless open-source solutions for this, but most of them fall short and/or add unnecessary obfuscation/complexity. We’ve been able to build our agentic flows using a combination of tool calling and JSON responses, but there’s still a missing higher order component that no one seems to have cracked yet.
phren0logy 2y ago
I'm a bit surprised at the approach to RAG. It will be great to see how well it handles complex PDFs. The max size is far larger than the Anthropic API permits (though that's obviously very different - no RAG). The chunking strategy is... pretty basic, but I guess we'll see if it works well enough for enough people.
cosbgn 2y ago
We handle over 1M requests per month using the Assistant API on https://rispose.com https://rispose.com which apparently will get depreciated mid 2026. Should we move to the new API?
- jstummbillig 2y ago
  Eventually, yes. The addressed Assistant API near the end of the the video: They say there will be a transition path, once they built all Assistant features into the new API, and ample time to take action.
- nknj 2y ago
  there's no rush to do this - in the coming weeks, we will add support for: - assistant-like and thread-like objects to the responses api - async responses - code interpreter in responses once we do this, we'll share a migration guide that allows you to move over without any loss of features or data. we'll also give you a full 12 months to do your migration. feel free to reach out at nikunj[at]openai.com if you have any questions about any of this, and thank you so much for building on the assistants api beta! I think you'll really like responses api too!
  - marko-k 2y ago
    If Responses is replacing Assistants, is there a quickstart template available—similar to the one you had for Assistants? https://github.com/openai/openai-assistants-quickstart https://github.com/openai/openai-assistants-quickstart
dmayle 2y ago
Is it just me, or is what OpenAI is really lacking is a billing API/platform? As an engineer, I have to manage the cost/service ratio manually, making sure I charge enough to handle my traffic, while enforcing/managing/policing the usage. Additionally, there are customers who already pay for OpenAI, so the value add for them is less, since they are paying twice for the underlying capabilities. If OpenAPI had a billing API/platform ala AppStore/PlayStore, I have multiple price points matched to OpenAI usage limits (and maybe configurable profit margins). For customers that don't have an existing relationship with me, OpenAI could support a Netflix/YouTube-style profit-sharing system, where OpenAI customers can try out and use products integrated with the billing platform/API, and my products would receive payment in accordance with customer usage...
- mrcwinn 2y ago
  One, if you charge above API costs, you should never police usage (so long as you're transparent with customers). Why would you need to cap usage if you're pricing correctly? (Rate limits aside) Two, yes, many people will pay $20/mo for ChatGPT and then also pay for a product that under the hood uses OpenAI API. If you're worried about your product's value not being differentiated from ChatGPT, I'd say you have a product problem moreso than OpenAI has a billing model problem.
- deleted 2y ago
  [deleted]
falcor84 2y ago
I'm impressed by the advances in Computer Use mentioned here and this got me wondering - is this already mature enough to be utilized for usability testing? Would I be right to assume that in general, a UI that is more difficult for AI to navigate is likely to also be relatively difficult for humans, and that it's a signal that it should be simplified/improved in some way?
- m3t4man 2y ago
  Why would you assume that? Modality of engagement is drastically different between the way LLM engages with UI vs human being
  - falcor84 2y ago
    Oh, I had assumed that it was trained on human interaction data and should be generally similar, and from the examples I saw - it generally was (although still not as good as us). In what sense do you expect it to be drastically different?
    - m3t4man 2y ago
      LLM engages with UI in a pragmatic way. It doesn't care about aesthetics or structural complexity, it doesn't care about latency or tactility. It doesnt have anywhere close to similar preferences and reasons for why it Vs a human being would and wouldn't click on or engage with something on a UI. Do you really see no difference in the way a hollow algorithm interacts with UI vs how a person would go about interacting with UI?
daviding 2y ago
It would have been nice if the Completions use of the internal web-search tool wasn't always mandatory and could be set to 'auto'. Would save a lot of reworking just to go the new Responses API format just for that use case.
theuppermiddle 2y ago
Does the SDK allow executing Python code generated in some sort of sandbox? If not are there any open source library which does this for us? I would ideally like the state of the code executed, including return values, available for the entire chat session, like IPython, so that subsequent LLM generated code can use them.
- sci_prog 2y ago
  Yeah, OpenInterpreter does this (you are not limited to OpenAI only): https://github.com/OpenInterpreter/open-interpreter https://github.com/OpenInterpreter/open-interpreter I wrote a wrapper around it that works in a web browser (you'll need an OpenAI API key): https://github.com/uhsealevelcenter/IDEA https://github.com/uhsealevelcenter/IDEA
nekitamo 2y ago
Does the new Agents SDK support streaming audio and Realtime models?
- rohanmehta1 2y ago
  Not yet, but it's on the roadmap!
simonw 2y ago
There's a really good thread on Twitter from the designer of the new APIs going into the background behind many of the design decisions: https://twitter.com/athyuttamre/status/1899541471532867821 https://twitter.com/athyuttamre/status/1899541471532867821 Here's the alternative link for people who aren't signed in to Twitter: https://nitter.net/athyuttamre/status/1899541471532867821 https://nitter.net/athyuttamre/status/1899541471532867821
- bradyriddle 2y ago
  The nitter link is appreciated!
  - ElijahLynn 2y ago
    TIL about Nitter, so grateful as I have Twitter blocked on my computer and phone.
    - sixhobbits 2y ago
      oh cool, I thought nitter died with the API changes. Glad they have it working again.
- telotortium 2y ago
  https://threadreaderapp.com/thread/1899541471532867821.html https://threadreaderapp.com/thread/1899541471532867821.html
cowpig 2y ago
Feels like OpenAI really want to compete with its own ecosystem. I guess they are doing this to try to position themselves as the standard web index that everyone uses, and the standard RAG service, etc. But they could just make great services and live in the infra layer instead of trying to squeeze everyone out at the application layer. Seems unnecessarily ecosystem-hostile
- pas 2y ago
  they target new entrants probably, they need more revenue, and more importantly a killer app that's at least a bit tied to them.
andrethegiant 2y ago
$25 per thousand searches seems excessive
- shrisukhani 2y ago
  ya i'm sure they'll get a bunch of usage despite that but don't know who would use it at any kind of scale with that pricing otoh, they've dropped prices for everything else a ton previously so maybe they will for this as well
- tiniuclx 2y ago
  Perplexity charges $5 per 1k searches for their Sonar API - this is pretty ridiculous.
simonw 2y ago
If you want to get an idea for the changes, here's a giant commit where they updated ALL of the Python library examples in one go from the old chat completions to the new resources APIs: https://github.com/openai/openai-python/commit/2954945ecc185259cfd7cd33c8cbc818a88e4e1b https://github.com/openai/openai-python/commit/2954945ecc185...
bob1029 2y ago
I don't know how much this API churn is going to help developers who are trying to integrate OAI into real, actual, non-wrapper products. Every vendor-managed state machine that handles conversation, messages, prompt hand-off, etc., has ultimately proven inadequate, presumptive or distracting for my use cases. At the end of the day, all I ever seem to use is the chat completion API with structured outputs turned on. Despite my "basic" usage, I am employing tool use, recursive conversations, RAG, etc. I don't see the value in outsourcing state management of my "agent" to a 3rd party. I have way more autonomy if I keep things like this local. The entire premise of these products is that you are feeding a string literal into some black box and it gives you a new string. Hopefully, as JSON or whatever you requested. If you focus just on the idea of composing the appropriate string each time, everything else melts away. This is the only grain that really matters. Think about other ways in which we compose highly-structured strings based upon business state stored in a database. It's literally the exact same thing you do when you SSR a webpage with PHP. The only real difference is how it is served.
- EGreg 2y ago
  “ These new tools streamline core agent logic, orchestration, and interactions, making it significantly easier for developers to get started with building agents” Sounds exactly like “the cloud”, especially AWS. Basically “get married to our platform, build on top of it, and make it hard to leave.” The benefits are that it’s easy to get started. And also that they invested in the infrastructure, but now they are trying to lock you in by storing as much state and data as possible with them withoit an easy way to migrate. So, increase your switching costs. For social networks the benefit was that they had the network effect but that doesn’t apply here.
  - Androider 2y ago
    All of AWS' big money makers are the meat-and-potatoes services around compute, storage, databases etc. where you could drop their offering and replace it with another in a straightforward way. It will cost you to migrate in terms of time and direct spend (those egress fees...), but it's possible. Companies ultimately stay put because the products work and the price is reasonable, but if they tried to 10X the price overnight everyone would eventually bolt. Yeah they keep pushing higher-level services, but the uptake of these is extremely limited. If you used something like SageMaker, which has an extremely high lock-in factor, it's probably because you're an old school company that don't know what you're doing and AWS held your developer's hand to get the Hello World-level app working, but at least you got your name printed in their case study materials of the project at the end. I think OpenAI looks at AWS and thinks they can do better. And for their investors, they must do better. But in the end I think the commoditization of LLMs is already almost complete, and this is just a futile attempt to fight it.
preaching5271 2y ago
When TypeScript SDK?
mrtksn 2y ago
I feel like all those AI agent attempts are misguided at their core because they don't attempt to create new ways but replace humans on the legacy systems. This is fundamentally shortsighted because the economy, life and everything is about humans interacting with humans. The current AI agent approach appears to be permutations of the joke about how people will make AI to expand their once sentence to a long nice e-mail and the AI on the receiving end will summarize that long e-mail into single sentence. I get that there's a use case for automating tasks on legacy systems but IMHO the real opportunity is the opportunity to remove most of the legacy systems. Humans are not that bad you know? Is it creating UIs for humans using AI then make AI use these UI to do stuff really the way forward?
- bob1029 2y ago
  I think the most valuable path for the current generation of AI models is integrating them with the configuration and administration side of the product. For example, as a supplemental user experience that power users in your org can leverage to macro out client configuration and project management tasks in a B2B SaaS ecosystem. Tool use can be very reliable when you have a well constrained set of abstractions, contexts and users to work with.
- NitpickLawyer 2y ago
  > because the economy, life and everything is about humans interacting with humans. How many hand crafted, clay bowls, baked in a human powered kiln are you using everyday? Or how many weaved baskets, made out of hand picked sticks? History has showed that anything that can be automated, will be automated. And everything that can be made "cheaper" or "faster" will as well.
  - mrtksn 2y ago
    That's not the point though, I'm not anti-automation or anything like that. The point is, using robots on interfaces and systems made for people is not the way to go. Why would you want to have your swipes on Tinder and your trip planning to Rio be automated through human interface? If it was for legit reasons it would have happened as from machine to machine communications. I'm big fan of the AI agent concept, my objection is that at in its current state people don't think out of the box and propose using the current infrastructure to delegate human functions instead of re-imagining the new world that is possible when working together with AI.
deleted 2y ago
[deleted]
huqedato 2y ago
OpenAI has become the Yahoo of the AI landscape.
shrisukhani 2y ago
Curious what people think about the CUA API pricing? Any thoughts on what use cases it may or may not work for (the pricing specifically)?
LeoPanthera 2y ago
I don't know about agents, but this finally adds the ability to search the web to the API. This is a very useful big deal. Kind of annoying that they've made a bunch of tiny changes to the history format though. It doesn't seem to change anything important, and only serves to make existing code incompatible.
dazzaji 2y ago
I was fortunate to get early access to the new Agent SDK and APIs that OpenAI dropped today and made an open source project to show some of the capabilities [1]. If you are using any of the other agent frameworks like LangGraph/LangChain, AutoGen, Crew, etc I definitely suggest giving this agent SDK a spin. To ease into it, I added the entire SDK with examples and full documentation as a single text file in my repo [2] so you can quickly get up to speed be adding it to a prompt and just asking about it or getting some quick start code to play around with. The code in my repo is very modular so you can try implementing any module using one of the other frameworks to do a head-to-head. Here’s a blog post with some more thoughts on this SDK [3] and some if its major capabilities. I’m liking it. A lot! [1] https://github.com/dazzaji/agento6 https://github.com/dazzaji/agento6 [2] https://raw.githubusercontent.com/dazzaji/agento6/refs/heads/main/openai_openai-agents-python.md https://raw.githubusercontent.com/dazzaji/agento6/refs/heads... [3] https://www.dazzagreenwood.com/p/unleashing-creativity-with-openais https://www.dazzagreenwood.com/p/unleashing-creativity-with-...
tiniuclx 2y ago
At $30 per 1k search queries, the OpenAI search API seems very expensive. Perplexity's Sonar model charges just $5 per thousand searches [0]. I wonder what justifies this drastic difference in price. [0] https://docs.perplexity.ai/guides/pricing https://docs.perplexity.ai/guides/pricing
casey2 2y ago
It's so obvious when you are just intentionally holding back releases just to steal mindshare from your competition. YAWN!
the_clarence 2y ago
Can you actually see what requests it makes to be able to answer?
sixhobbits 2y ago
Aa lot of criticism about the potential of vendor lockin etc, but I think this is great, especially for building proof of concepts and small projects. As they said, these are the first building blocks and they look great to me. When gpt3.5 came out these are literally the first things I built manually. I mainly use LLMs through a telegram bot. I know there are a lot of tools and frameworks out there but I wrote a few hundred lines of hacky python to give my bot memory, web search, image analysis. It's fun and useful and I agree that these are the basic building blocks that many apps need. Sure you can find better stuff elsewhere with less lock in and more control, but now it "just works" and this responses api is cleaner and more powerful than the chatcompletions one, so personally I'm happy to give openai credit for this, I just don't know why they couldn't have released it two years ago
laacz 2y ago
Won't the industry eventually go through all stages of the smart home? Early protocols, often completely incompatible and perpendicular, then few competing standards, etc?

389 points by meetpateltech 2y ago

hodanli 2y ago

I wonder why they phased out Pydantic in structured output for the Responses API.

sdcoffey 2y ago
Hey there! This is Steve here from the OpenAI team–I worked on the Responses API. We have not removed this! It should still work just like before! Here's an example: https://github.com/openai/openai-python/blob/main/examples/responses/structured_outputs.py https://github.com/openai/openai-python/blob/main/examples/r...
- hodanli 2y ago
  Hi Steve, I based my assumption on the examples in the documentation[^1]. It is great that we can still use that. 1. https://platform.openai.com/docs/guides/structured-outputs?api-mode=responses https://platform.openai.com/docs/guides/structured-outputs?a...
lunarcave 2y ago
(Shameless plug) I worked on something for anyone else wanting to get structured outputs from LLMs in a model agnostic way (Including Open AI models): https://github.com/inferablehq/l1m https://github.com/inferablehq/l1m

nowittyusername 2y ago

How does this compare to MCP? Anyone has any considerations on the matter?

nnurmanov 2y ago

Does anyone know if there are any difference if you typed the question with typos vs you did it correctly?

davidbarker 2y ago
In theory there shouldn't be — LLMs are pretty robust to typos and usually infer the intended meaning regardless.

swyx 2y ago

swyx here. we got some preview and time with the API/DX team to ask FAQs about all the new APIs. https://latent.space/p/openai-agents-platform https://latent.space/p/openai-agents-platform main fun part - since responses are stored for free by default now, how can we abuse the Responses API as a database :) other fun qtns that a HN crew might enjoy: - hparams for websearch - depth/breadth of search for making your own DIY Deep Research - now that OAI is offering RAG/reranking out of the box as part of the Responses API, when should you build your own RAG? (i basically think somebody needs to benchmark the RAG capabilities of the Files API now, because the community impression has not really updated from back when Assistants API was first launched) - whats the diff between Agents SDK and OAI Swarm? (basically types, tracing, pluggable LLMs) - will the `search-preview` and `computer-use-preview` finetunes be merged into GPT5?

ggnore7452 2y ago
appreciate the question on hparams for websearch! one of the main reasons i build these ai search tools from scratch is that i can fully control the depth and breadth (and also customize loader to whatever data/sites). and currently the web search isn't very transparent on what sites they do not have full text or just use snippets. having computer use + websearch is definitely something very powerful (openai's deep research essentially)
mritchie712 2y ago
for anyone that likes the Agents SDK, but doesn't want their framework attached to OpenAI, we're really liking PydanticAI[0]. 0 - https://ai.pydantic.dev/ https://ai.pydantic.dev/
- fullstackwife 2y ago
  Openai SDK docs: > Notably, our SDK is compatible with any model providers that support the OpenAI Chat Completions API format. so you can use with everything, not only OpenAI?
  - DrBenCarson 2y ago

baxtr 2y ago

A bit off topic but the post comes handy: can we settle the debate what an agent really is? It seems like everyone has their own definition. Ok I’ll start: an agent is a computer program that utilized LLMs heutiger for decision making.

codydkdc 2y ago
an agent is software that does something on behalf of someone (aka software) I personally strongly prefer the term "bots" for what most of these frameworks call "agents"
- handfuloflight 2y ago
  Stick to the agentic nomenclature if you want at least a magnitude increase in valuation.
deleted 2y ago
[deleted]
3stripe 2y ago
First rule of writing definitions: use everyday English.
- baxtr 2y ago
  True! Meant heuristic
knowaveragejoe 2y ago
I think Anthropic's definition makes the most sense. - Workflows are systems where LLMs and tools are orchestrated through predefined code paths. (imo this is what most people are referring to as "agents") - Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks. https://www.anthropic.com/engineering/building-effective-agents https://www.anthropic.com/engineering/building-effective-age...

zellyn 2y ago

Notably not mentioned: Model Context Protocol https://www.anthropic.com/news/model-context-protocol https://www.anthropic.com/news/model-context-protocol

esafak 2y ago
How do they compare?
- cowpig 2y ago
  MCP is a protocol, and Anthropic has provided SDKs for implementing that protocol. In practice, I find the MCP protocol to be pretty great, but it leaves basically everything except the model parts out. I.e. MCP really only addresses how "agentic" systems interact with one another, nothing else. This SDK is trying to provide a bunch of code for implementing specific agent codebases. There are a bunch of open source ones already, so this is OpenAI throwing their hat in the ring. IMO this OpenAI release is kind of ecosystem-hostile in that they are directly competing with their users, in the same way that the GPT apps were.
  - esafak 2y ago
    Thank you. Which open source ones do you recommend?
    - cowpig 2y ago
      Here's a fairly comprehensive list: https://github.com/slavakurilyak/awesome-ai-agents https://github.com/slavakurilyak/awesome-ai-agents CrewAI is a popular VC-backed one, but two that I think are kind of interesting in the open source space are: https://github.com/i-am-bee/beeai-framework https://github.com/i-am-bee/beeai-framework https://github.com/lastmile-ai/mcp-agent https://github.com/lastmile-ai/mcp-agent ... However I think the vast majority of "AI Agent" use-cases in practice right now are actually just workflows, and imo dify is great for those: https://github.com/langgenius/dify https://github.com/langgenius/dify [edit] worth mentioning [langfuse](https://github.com/langfuse/langfuse https://github.com/langfuse/langfuse), which is more like a platform that addresses the observability/evals/prompt management piece of the puzzle as opposed to a full-on "agent framework". In practice I have not yet run into a case where I needed something like what OpenAI just released, nor crewAI etc (despite it feeling like those cases may be coming)

rvz 2y ago

They did not announce the price(s) in the presentation. Likely because they know it is going to be very expensive: Web Search [0] * $30 and $25 per 1K queries for GPT‑4o search and 4o-mini search. File search [1] * $2.50 per 1K queries and file storage at $0.10/GB/day * First 1GB is free. Computer use tool (computer-use-preview model) [2] * $3 per 1M input tokens and $12/1M output tokens. [0] https://platform.openai.com/docs/pricing#web-search https://platform.openai.com/docs/pricing#web-search [1] https://platform.openai.com/docs/pricing#built-in-tools https://platform.openai.com/docs/pricing#built-in-tools [2] https://platform.openai.com/docs/pricing#latest-models https://platform.openai.com/docs/pricing#latest-models

yard2010 2y ago
So they're basically pivoting from selling text by the ounce to selling web searches and cloud storage? I like it, it's a bold move. When the slow people at Google finally catch up it might be too late for Google?
- KoolKat23 2y ago
  Google AI Studios "Grounding" basically web search is priced similarly. (Very expensive for either, although Google gives you your first 1500 queries free). It seems completely upside down, they always said traditional search was cheaper/less intensive, I guess a lot of tokens must go into the actual LLM searching and retrieving.
rudedogg 2y ago
Those prices, especially for web search, are absurd. I have a hard time seeing how this API is better than https://www.anthropic.com/news/model-context-protocol https://www.anthropic.com/news/model-context-protocol. It seems like the motivation was "how can we make more money", rather than "how can we be more useful for our users".
_bramses 2y ago
For anyone who's looking Brave Search is $3/1000 requests [0] I also wrote a script that searches the web and works pretty well (using the vercel ai sdk)[1] [0] - https://brave.com/search/api/ https://brave.com/search/api/ [1] - https://gist.github.com/bramses/41e90b27d156590154bcefd4119f2b19 https://gist.github.com/bramses/41e90b27d156590154bcefd4119f...

Areibman 2y ago

Nice to finally see one of the labs throwing weight behind a much needed simple abstraction. It's clear they learned from the incumbents (langchain et al)-- don't sell complexity. Also very nice of them to include extensible tracing. The AgentOps integration is a nice touch to getting behind the scenes to understand how handoffs and tool calls are triggered

swyx 2y ago
why agentops specifically? doesnt the oai first party one also do it?
- Areibman 2y ago
  The OpenAI dash is great but is clearly missing a lot of features (i.e. data export, alerts, non-oai model compatibility). Believe it or not, but they don't even report response API costs on spans
  - swyx 2y ago
    oh lol i wasnt looking at usernames. it's you! spiderman pointing
esafak 2y ago
Extensible how?
deleted 2y ago
[deleted]

serjester 2y ago

This is one of the few agent abstractions I've seen that actually seems intuitive. Props to the OpenAI team, seems like it'll kill a lot of bad startups.

sdcoffey 2y ago
Steve here from the OpenAI team–this means a lot! We really hope you enjoy building on it

ilaksh 2y ago

The Agents SDK they linked to comes up 404. BTW I have something somewhat similar to some of this like Responses and File Search in MindRoot by using the task API: https://github.com/runvnc/mindroot/blob/main/api.md https://github.com/runvnc/mindroot/blob/main/api.md Which could be combined with the query_kb tool from the mr_kb plugin (in my mr_kb repo) which is actually probably better than File Search because it allows searching multiple KBs. Anyway, if anyone wants to help with my program, create a plugin on PR, or anything, feel free to connect on GitHub, email or Discord/Telegram (runvnc).

yablak 2y ago
Loads fine for me. Maybe because I'm logged in?
- IncreasePosts 2y ago
  That should be a 403 then. Tsk tsk open ai
  - 29ebJCyy 2y ago
    Technically it should be a 401. Tsk tsk IncreasePosts.
    - __float 2y ago
      It's common (see: S3, private GitHub repos) to return 404 instead of unauthorized to avoid even leaking existence of a resource at URL.
edwinarbus 2y ago
sorry, fixed link: https://openai.github.io/openai-agents-python/ https://openai.github.io/openai-agents-python/

anorak27 2y ago

I have built myself a much simpler and powerful version of the responses API and it works with all LLM providers. https://github.com/Anilturaga/aiide https://github.com/Anilturaga/aiide

grvdrm 2y ago
Thank you for your detailed Readme. A relief / joy to read compared to many other libraries/etc. that provide one basic (if that) example and otherwise leave to your own trial/error.
bsenftner 2y ago
For so few Github stars, I'm surprised that this is the 4th time I'm reading about your aiide project in 2 days. It looks good, very good BTW.

nextworddev 2y ago

This may be bad for Langflow, Langsmith, etc

mentalgear 2y ago

Well, I'll just wait 2-3 days until a (better) open-source alternative is released. :D

jumploops 2y ago

> “we plan to formally announce the deprecation of the Assistants API with a target sunset date in mid-2026.” The new Responses API is a step in the right direction, especially with the built-in “handoff” functionality. For agentic use cases, the new API still feels a bit limited, as there’s a lack of formal “guardrails”/state machine logic built in. > “Our goal is to give developers a seamless platform experience for building agents” It will be interesting to see how they move towards this platform, my guess is that we’ll see a graph-based control flow in the coming months. Now there are countless open-source solutions for this, but most of them fall short and/or add unnecessary obfuscation/complexity. We’ve been able to build our agentic flows using a combination of tool calling and JSON responses, but there’s still a missing higher order component that no one seems to have cracked yet.

phren0logy 2y ago

I'm a bit surprised at the approach to RAG. It will be great to see how well it handles complex PDFs. The max size is far larger than the Anthropic API permits (though that's obviously very different - no RAG). The chunking strategy is... pretty basic, but I guess we'll see if it works well enough for enough people.

cosbgn 2y ago

We handle over 1M requests per month using the Assistant API on https://rispose.com https://rispose.com which apparently will get depreciated mid 2026. Should we move to the new API?

jstummbillig 2y ago
Eventually, yes. The addressed Assistant API near the end of the the video: They say there will be a transition path, once they built all Assistant features into the new API, and ample time to take action.
nknj 2y ago
there's no rush to do this - in the coming weeks, we will add support for: - assistant-like and thread-like objects to the responses api - async responses - code interpreter in responses once we do this, we'll share a migration guide that allows you to move over without any loss of features or data. we'll also give you a full 12 months to do your migration. feel free to reach out at nikunj[at]openai.com if you have any questions about any of this, and thank you so much for building on the assistants api beta! I think you'll really like responses api too!
- marko-k 2y ago
  If Responses is replacing Assistants, is there a quickstart template available—similar to the one you had for Assistants? https://github.com/openai/openai-assistants-quickstart https://github.com/openai/openai-assistants-quickstart

dmayle 2y ago

Is it just me, or is what OpenAI is really lacking is a billing API/platform? As an engineer, I have to manage the cost/service ratio manually, making sure I charge enough to handle my traffic, while enforcing/managing/policing the usage. Additionally, there are customers who already pay for OpenAI, so the value add for them is less, since they are paying twice for the underlying capabilities. If OpenAPI had a billing API/platform ala AppStore/PlayStore, I have multiple price points matched to OpenAI usage limits (and maybe configurable profit margins). For customers that don't have an existing relationship with me, OpenAI could support a Netflix/YouTube-style profit-sharing system, where OpenAI customers can try out and use products integrated with the billing platform/API, and my products would receive payment in accordance with customer usage...

mrcwinn 2y ago
One, if you charge above API costs, you should never police usage (so long as you're transparent with customers). Why would you need to cap usage if you're pricing correctly? (Rate limits aside) Two, yes, many people will pay $20/mo for ChatGPT and then also pay for a product that under the hood uses OpenAI API. If you're worried about your product's value not being differentiated from ChatGPT, I'd say you have a product problem moreso than OpenAI has a billing model problem.
deleted 2y ago
[deleted]

falcor84 2y ago

I'm impressed by the advances in Computer Use mentioned here and this got me wondering - is this already mature enough to be utilized for usability testing? Would I be right to assume that in general, a UI that is more difficult for AI to navigate is likely to also be relatively difficult for humans, and that it's a signal that it should be simplified/improved in some way?

m3t4man 2y ago
Why would you assume that? Modality of engagement is drastically different between the way LLM engages with UI vs human being
- falcor84 2y ago
  Oh, I had assumed that it was trained on human interaction data and should be generally similar, and from the examples I saw - it generally was (although still not as good as us). In what sense do you expect it to be drastically different?
  - m3t4man 2y ago
    LLM engages with UI in a pragmatic way. It doesn't care about aesthetics or structural complexity, it doesn't care about latency or tactility. It doesnt have anywhere close to similar preferences and reasons for why it Vs a human being would and wouldn't click on or engage with something on a UI. Do you really see no difference in the way a hollow algorithm interacts with UI vs how a person would go about interacting with UI?

daviding 2y ago

It would have been nice if the Completions use of the internal web-search tool wasn't always mandatory and could be set to 'auto'. Would save a lot of reworking just to go the new Responses API format just for that use case.

theuppermiddle 2y ago

Does the SDK allow executing Python code generated in some sort of sandbox? If not are there any open source library which does this for us? I would ideally like the state of the code executed, including return values, available for the entire chat session, like IPython, so that subsequent LLM generated code can use them.

sci_prog 2y ago
Yeah, OpenInterpreter does this (you are not limited to OpenAI only): https://github.com/OpenInterpreter/open-interpreter https://github.com/OpenInterpreter/open-interpreter I wrote a wrapper around it that works in a web browser (you'll need an OpenAI API key): https://github.com/uhsealevelcenter/IDEA https://github.com/uhsealevelcenter/IDEA

nekitamo 2y ago

Does the new Agents SDK support streaming audio and Realtime models?

rohanmehta1 2y ago
Not yet, but it's on the roadmap!

simonw 2y ago

There's a really good thread on Twitter from the designer of the new APIs going into the background behind many of the design decisions: https://twitter.com/athyuttamre/status/1899541471532867821 https://twitter.com/athyuttamre/status/1899541471532867821 Here's the alternative link for people who aren't signed in to Twitter: https://nitter.net/athyuttamre/status/1899541471532867821 https://nitter.net/athyuttamre/status/1899541471532867821

bradyriddle 2y ago
The nitter link is appreciated!
- ElijahLynn 2y ago
  TIL about Nitter, so grateful as I have Twitter blocked on my computer and phone.
  - sixhobbits 2y ago
    oh cool, I thought nitter died with the API changes. Glad they have it working again.
telotortium 2y ago
https://threadreaderapp.com/thread/1899541471532867821.html https://threadreaderapp.com/thread/1899541471532867821.html

cowpig 2y ago

Feels like OpenAI really want to compete with its own ecosystem. I guess they are doing this to try to position themselves as the standard web index that everyone uses, and the standard RAG service, etc. But they could just make great services and live in the infra layer instead of trying to squeeze everyone out at the application layer. Seems unnecessarily ecosystem-hostile

pas 2y ago
they target new entrants probably, they need more revenue, and more importantly a killer app that's at least a bit tied to them.

andrethegiant 2y ago

$25 per thousand searches seems excessive

shrisukhani 2y ago
ya i'm sure they'll get a bunch of usage despite that but don't know who would use it at any kind of scale with that pricing otoh, they've dropped prices for everything else a ton previously so maybe they will for this as well
tiniuclx 2y ago
Perplexity charges $5 per 1k searches for their Sonar API - this is pretty ridiculous.

simonw 2y ago

If you want to get an idea for the changes, here's a giant commit where they updated ALL of the Python library examples in one go from the old chat completions to the new resources APIs: https://github.com/openai/openai-python/commit/2954945ecc185259cfd7cd33c8cbc818a88e4e1b https://github.com/openai/openai-python/commit/2954945ecc185...

bob1029 2y ago

I don't know how much this API churn is going to help developers who are trying to integrate OAI into real, actual, non-wrapper products. Every vendor-managed state machine that handles conversation, messages, prompt hand-off, etc., has ultimately proven inadequate, presumptive or distracting for my use cases. At the end of the day, all I ever seem to use is the chat completion API with structured outputs turned on. Despite my "basic" usage, I am employing tool use, recursive conversations, RAG, etc. I don't see the value in outsourcing state management of my "agent" to a 3rd party. I have way more autonomy if I keep things like this local. The entire premise of these products is that you are feeding a string literal into some black box and it gives you a new string. Hopefully, as JSON or whatever you requested. If you focus just on the idea of composing the appropriate string each time, everything else melts away. This is the only grain that really matters. Think about other ways in which we compose highly-structured strings based upon business state stored in a database. It's literally the exact same thing you do when you SSR a webpage with PHP. The only real difference is how it is served.

EGreg 2y ago
“ These new tools streamline core agent logic, orchestration, and interactions, making it significantly easier for developers to get started with building agents” Sounds exactly like “the cloud”, especially AWS. Basically “get married to our platform, build on top of it, and make it hard to leave.” The benefits are that it’s easy to get started. And also that they invested in the infrastructure, but now they are trying to lock you in by storing as much state and data as possible with them withoit an easy way to migrate. So, increase your switching costs. For social networks the benefit was that they had the network effect but that doesn’t apply here.
- Androider 2y ago
  All of AWS' big money makers are the meat-and-potatoes services around compute, storage, databases etc. where you could drop their offering and replace it with another in a straightforward way. It will cost you to migrate in terms of time and direct spend (those egress fees...), but it's possible. Companies ultimately stay put because the products work and the price is reasonable, but if they tried to 10X the price overnight everyone would eventually bolt. Yeah they keep pushing higher-level services, but the uptake of these is extremely limited. If you used something like SageMaker, which has an extremely high lock-in factor, it's probably because you're an old school company that don't know what you're doing and AWS held your developer's hand to get the Hello World-level app working, but at least you got your name printed in their case study materials of the project at the end. I think OpenAI looks at AWS and thinks they can do better. And for their investors, they must do better. But in the end I think the commoditization of LLMs is already almost complete, and this is just a futile attempt to fight it.

preaching5271 2y ago

When TypeScript SDK?

mrtksn 2y ago

I feel like all those AI agent attempts are misguided at their core because they don't attempt to create new ways but replace humans on the legacy systems. This is fundamentally shortsighted because the economy, life and everything is about humans interacting with humans. The current AI agent approach appears to be permutations of the joke about how people will make AI to expand their once sentence to a long nice e-mail and the AI on the receiving end will summarize that long e-mail into single sentence. I get that there's a use case for automating tasks on legacy systems but IMHO the real opportunity is the opportunity to remove most of the legacy systems. Humans are not that bad you know? Is it creating UIs for humans using AI then make AI use these UI to do stuff really the way forward?

bob1029 2y ago
I think the most valuable path for the current generation of AI models is integrating them with the configuration and administration side of the product. For example, as a supplemental user experience that power users in your org can leverage to macro out client configuration and project management tasks in a B2B SaaS ecosystem. Tool use can be very reliable when you have a well constrained set of abstractions, contexts and users to work with.
NitpickLawyer 2y ago
> because the economy, life and everything is about humans interacting with humans. How many hand crafted, clay bowls, baked in a human powered kiln are you using everyday? Or how many weaved baskets, made out of hand picked sticks? History has showed that anything that can be automated, will be automated. And everything that can be made "cheaper" or "faster" will as well.
- mrtksn 2y ago
  That's not the point though, I'm not anti-automation or anything like that. The point is, using robots on interfaces and systems made for people is not the way to go. Why would you want to have your swipes on Tinder and your trip planning to Rio be automated through human interface? If it was for legit reasons it would have happened as from machine to machine communications. I'm big fan of the AI agent concept, my objection is that at in its current state people don't think out of the box and propose using the current infrastructure to delegate human functions instead of re-imagining the new world that is possible when working together with AI.

deleted 2y ago

[deleted]

huqedato 2y ago

OpenAI has become the Yahoo of the AI landscape.

shrisukhani 2y ago

Curious what people think about the CUA API pricing? Any thoughts on what use cases it may or may not work for (the pricing specifically)?

LeoPanthera 2y ago

I don't know about agents, but this finally adds the ability to search the web to the API. This is a very useful big deal. Kind of annoying that they've made a bunch of tiny changes to the history format though. It doesn't seem to change anything important, and only serves to make existing code incompatible.

dazzaji 2y ago

I was fortunate to get early access to the new Agent SDK and APIs that OpenAI dropped today and made an open source project to show some of the capabilities [1]. If you are using any of the other agent frameworks like LangGraph/LangChain, AutoGen, Crew, etc I definitely suggest giving this agent SDK a spin. To ease into it, I added the entire SDK with examples and full documentation as a single text file in my repo [2] so you can quickly get up to speed be adding it to a prompt and just asking about it or getting some quick start code to play around with. The code in my repo is very modular so you can try implementing any module using one of the other frameworks to do a head-to-head. Here’s a blog post with some more thoughts on this SDK [3] and some if its major capabilities. I’m liking it. A lot! [1] https://github.com/dazzaji/agento6 https://github.com/dazzaji/agento6 [2] https://raw.githubusercontent.com/dazzaji/agento6/refs/heads/main/openai_openai-agents-python.md https://raw.githubusercontent.com/dazzaji/agento6/refs/heads... [3] https://www.dazzagreenwood.com/p/unleashing-creativity-with-openais https://www.dazzagreenwood.com/p/unleashing-creativity-with-...

tiniuclx 2y ago

At $30 per 1k search queries, the OpenAI search API seems very expensive. Perplexity's Sonar model charges just $5 per thousand searches [0]. I wonder what justifies this drastic difference in price. [0] https://docs.perplexity.ai/guides/pricing https://docs.perplexity.ai/guides/pricing

casey2 2y ago

It's so obvious when you are just intentionally holding back releases just to steal mindshare from your competition. YAWN!

the_clarence 2y ago

Can you actually see what requests it makes to be able to answer?

sixhobbits 2y ago

Aa lot of criticism about the potential of vendor lockin etc, but I think this is great, especially for building proof of concepts and small projects. As they said, these are the first building blocks and they look great to me. When gpt3.5 came out these are literally the first things I built manually. I mainly use LLMs through a telegram bot. I know there are a lot of tools and frameworks out there but I wrote a few hundred lines of hacky python to give my bot memory, web search, image analysis. It's fun and useful and I agree that these are the basic building blocks that many apps need. Sure you can find better stuff elsewhere with less lock in and more control, but now it "just works" and this responses api is cleaner and more powerful than the chatcompletions one, so personally I'm happy to give openai credit for this, I just don't know why they couldn't have released it two years ago

laacz 2y ago

Won't the industry eventually go through all stages of the smart home? Early protocols, often completely incompatible and perpendicular, then few competing standards, etc?