6 ms·

A Survey of AI Agent Protocols

91 points by distalx 1y ago

simonw 1y ago
I scrolled straight to section 2.1 "Definition and Characteristics of LLM agents" to find out which of the many definitions of "agent" they are using here. They went for LLM + short-term and long-term memory + planning + tool using + action execution. Presumably "planning" here is covered by any LLM that can do "think step by step" reasonably well? It wasn't clear to me what the difference between "tool using" and "action execution" was. I haven't seen a definition that specifically encompasses both short- and long-term memory before. They say: > This dual memory system allows agents to maintain conversation continuity while building knowledge over time. So presumably, this is the standard LLM chat conversation log plus a tool that can decide to stash extra information in a permanent store - similar to how ChatGPT's memory feature worked up until about four weeks ago.
- swyx 1y ago
  > Presumably "planning" here is covered by any LLM that can do "think step by step" reasonably well? mild disagree. 1) externalizing the plan and letting the user audit/edit the plan while its working is "tool use", yes, but a very specialcase kind of tool use that, for example, operator and deep research use Temporal for. ofc we also saw this with Devin/Manus and i kinda think they're better 2) there is a form of primitive tree search that people are doing where they can spam out several different paths and run it a few steps ahead to gain information about optimal planning. You will see this with morph's launch at AIE. 3) plan meta reflection and reuse - again a form of tool use, but the devin and allhands folks have worked on this a lot more than most. my criticism of many agent definitions is that they generally do not take memory, planning, and auth seriously enough, and i think those 3 areas are my current bets for "alpha" in 2025. > I haven't seen a definition that specifically encompasses both short- and long-term memory before. here - https://docs.mem0.ai/core-concepts/memory-types#short-term-memory https://docs.mem0.ai/core-concepts/memory-types#short-term-m... - https://x.com/swyx/status/1915128966203236571 https://x.com/swyx/status/1915128966203236571
  - Tokumei-no-hito 1y ago
    what is morphs launch at AIE?
    - deleted 1y ago
      [deleted]
    - deleted 1y ago
      [deleted]
    - nkko 1y ago
      Morph.so is an ai sandbox thing that has branching from state, prob doing a talk in SF on this at AI Engineer conf
      - Tokumei-no-hito 1y ago
        thanks
- tough 1y ago
  > It wasn't clear to me what the difference between "tool using" and "action execution" was. doing a lot of inference here, but could be a separation between -read- tool kinda actions, and -write/execute- (like running code/sending an email, etc) a bit weird from a coding perspective but idk
- sebastiennight 1y ago
  > similar to how ChatGPT's memory feature worked up until about four weeks ago What happened four weeks ago?
  - swyx 1y ago
    they updated it opaquely - we no longer know anything about how it works https://www.latent.space/p/clippy-v-anton https://www.latent.space/p/clippy-v-anton
    - wunderwuzzi23 1y ago
      See my response to Simon above on some insights on how it works - I'll write up in detail in a blog post also when I get to it.
  - simonw 1y ago
    They added a feature where it can reference content from any conversation you've had with it in the past. It's not clear how this works - probably some kind of vector search? Prior to that change "memory" was a tool call: https://simonwillison.net/2024/Oct/15/chatgpt-horoscopes/#how-memory-works https://simonwillison.net/2024/Oct/15/chatgpt-horoscopes/#ho...
    - wunderwuzzi23 1y ago
      Here is what I have been able to reverse engineer for o3... At high level it maintains about ~40 conversations in system prompt under a section called "recent conversation content". It only contains what the user typed, not assistant responses (probably due to prompt injection) - there a few corner cases though. :) There are other sections in the system prompt now that contain aggregated info, so recent conversations turn into user insights over time I believe. It can't actually "search" history afaik - that part I'm still wondering, as it was my first thought on how it might work... I also found a way to exfiltrate the recent content - so hopefully that will be fixed soon... Overall, this feature creates a lot of confusion and response quality declines at times too - and anything someone posts now online (like weird behavior or hallucinations,...) is likely influenced by their past conversations! So it will make it more difficult to understand what's really happening. I think it would be cool if "projects" would be entirely isolated with their own memories and history etc. or have different "profiles"
- jsemrau 1y ago
  Accurate memory access across memory types is still not really solved. That is the issue. Most agent frameworks from the main model providers are still quite spotty. Image of a table outlining where the major frameworks are: https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76107516-eba3-4a17-ba85-5ccd76180d29_960x540.png https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_... Here is also an article I wrote last year on the different types of memory: https://open.substack.com/pub/jdsemrau/p/memory-and-knowledge-for-agents?r=7fx0t&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/jdsemrau/p/memory-and-knowledg...
anonymousDan 1y ago
The hype around agent protocols reminds me of the emperor's new clothes. There's just nothing to it from a technical perspective.
- esafak 1y ago
  It's just something that needs to be done if you want LLMs to be useful. Everything does not have to be a technological marvel.
- hendler 1y ago
  HTML was XML for the web. Nothing to it from a technical perspective.
  - pyuser583 1y ago
    XHTML was supposed to be revolutionary.
  - padolsey 1y ago
    HTML was designed prior to XML fwiw, spinning off from SGML.
- 01100011 1y ago
  Is it something like this: Ages and ages ago I was an EE turned programmer and everyone was hyping JUnit at the time. I had a customer ask for it on a project so fine I'll learn it. I kept thinking it was stupid because in my mind it barely did anything. But then I got it: it did barely do anything, but it did things you'd commonly need to do for testing and it did them in a somewhat standardized way that kept you from having to roll your own every time. Suddenly it didn't feel so stupid.
- crystal_revenge 1y ago
  Even worse is that virtually no one in this space even recognizes the fairly rich research into “agents” throughout the history of computer science. Mention John Holland’s work in adaptive systems, Hewitt’s actor model or even Minsky’s “Society of the Mind” and you’ll be met with blank stares. I do believe LLMs have the potential to make these older ideas relevant again and potentially create something amazing, but sadly the ignorant hype makes it virtually impossible to have intelligent conversations about these possibilities.
revskill 1y ago
Is AI Agent just LLM wrapper ? Is there anything more interesting to it ?
- mindcrime 1y ago
  I hate to say "it depends" but it, aaah, kinda depends. Nailing down a good definition of "Agent" has a been a problem dating back at least into the 1990's if not the 1980's. So, depending on which definition of "AI Agent" you're using, you arguably don't even need any LLM at all. Heck, using the most expansive definition I've seen, a mechanical thermostat counts. I don't know that I'd go that far, but I'll definitely say that I do not consider Agents to require use of LLM's. That said, the "Agent pattern du jour" is heavily based on using LLM's to provide the "brain" of the Agent and then Tool Calling to let it do things an LLM can't normally do. But still... depending on just what you do with those tool calls and any other code that sits in your Agent implementation then it certainly could be more than "just" an LLM wrapper. Nothing stops you from, for example, using the BDI architecture, implementing multi-level memory that's analogous to the way human memory works, wiring in some inductive learning, and throwing in some case-based reasoning, and an ontology based reasoning engine. Most people today aren't doing this, because they're mostly johnny-come-lately's that don't know anything about AI besides what they see on Twitter, Reddit, and LinkedIn; and wouldn't know BDI from BDSM. [1]: https://en.wikipedia.org/wiki/Belief%E2%80%93desire%E2%80%93intention_software_model https://en.wikipedia.org/wiki/Belief%E2%80%93desire%E2%80%93...
  - freeamz 1y ago
    Agent seems like a process/worker/thread that is running LLM inference?
- namaria 1y ago
  Yes. Check the codebases. It's all prompting scaffolding. All of it. Chain of though, agents, tool using. It's just parsing the user inputs, adding text around it. "You are an expert X". That's the whole edifice.
- MichaelMoser123 1y ago
sgt101 1y ago
Doesn't even mention KQML or FIPA - staggeringly ignorant.
- eMPee584 1y ago
  any experience with it? illuminate us lot if you will..
  - sgt101 1y ago
    A very old report about FIPA that was rejected from AAMAS. https://github.com/sgt101/FIPA-Mirror-/blob/main/Experience_report_paper.pdf https://github.com/sgt101/FIPA-Mirror-/blob/main/Experience_...
- ursaguild 1y ago
  I was under the impression ACL supersedes KQML and was proposed by FIPA?
  - sgt101 1y ago
    FIPA came after KQML and tried to learn from it and do something better. There were people who didn't think that it had succeeded in that. So it's debatable I think.
- dang 1y ago
  "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "When disagreeing, please reply to the argument instead of calling names. 'That is idiotic; 1 + 1 is 2, not 3' can be shortened to '1 + 1 is 2, not 3." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you know more than others, that's great, but in that case please share some of what you know, so the rest of us can learn. Putdowns don't help. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
arresin 1y ago
This is a very weird paper. They go through all these protocols yet haven’t don’t mention the most promising one (smolagents): https://github.com/huggingface/smolagents https://github.com/huggingface/smolagents
- ursaguild 1y ago
  smolagents by huggingface would be more of an agent framework. If it was discussed we would see smolagents/llamaindex/pydantic/etc with frameworks on figure 2. Several frameworks were left off in this paper as it focuses more on the protocols. https://huggingface.co/docs/smolagents/ https://huggingface.co/docs/smolagents/
  - arresin 1y ago
    Good point. Appreciate the correction.
klabb3 1y ago
It’s good to see a new fresh push for protocols and interop. However, I don’t think it will do the thing we hope for, for non-technical reasons. During Web 2.0, we saw similar enthusiasm. Instead of AI agents or blockchain, every modern company had an API exposed. For instance, Gmail- and Facebook chat was usable with 3p client apps. What killed this was not tech, but business. The product wasn’t say social media, it was ad delivery. And using APIs was considered a bypass of funnels that they want to control. Today, if you go to a consumer service website, you will generally be met with a login/app wall. Even companies that charge money directly (say 23&me ad an egregious example) are also data hoarders. Apple is probably a better example. There’s no escape. The point is, protocols is the easy part. If the economics and incentives are the same as yesterday, we will see similar outcomes. Today, the consumer web is adversarial between provider ”platforms”, ad delivery, content creators, and the products themselves (ie the people who use them).
- lorepieri 1y ago
  I really like this analysis. But what about companies allowing agents to interact natively (via API or similar) getting more of the agents inbound since agents are more optimised to go there? If people want to use agents it will cause a lot of lost revenue for companies not allowing agents to interact natively.
  - klabb3 1y ago
    It’s only lost revenue if the interactions were providing revenue in the first place. And for consumer tech, if you take away the ability to control the ad-funnels and engagement knobs, there is no revenue to gain. Why do you think all APIs got locked down? Nowadays even pure html delivery is getting locked down to prevent scraping. You think the SEO optimized recipe click holes are incentivized to deliver the content without the ads? I mean, it’s not impossible that consumers will ignore sites and services that don’t play with their AI. But even so, the content providers would need to find alternative revenue streams – otherwise they’d be bleeding money. My prediction is that tech companies will continue to compete for dominance over the entry points, apps and human interactions. It will be an adversarial space where coalitions of vertically integrated walled gardens can work. Basically how it is already.

91 points by distalx 1y ago

I scrolled straight to section 2.1 "Definition and Characteristics of LLM agents" to find out which of the many definitions of "agent" they are using here. They went for LLM + short-term and long-term memory + planning + tool using + action execution. Presumably "planning" here is covered by any LLM that can do "think step by step" reasonably well? It wasn't clear to me what the difference between "tool using" and "action execution" was. I haven't seen a definition that specifically encompasses both short- and long-term memory before. They say: > This dual memory system allows agents to maintain conversation continuity while building knowledge over time. So presumably, this is the standard LLM chat conversation log plus a tool that can decide to stash extra information in a permanent store - similar to how ChatGPT's memory feature worked up until about four weeks ago.

swyx 1y ago
> Presumably "planning" here is covered by any LLM that can do "think step by step" reasonably well? mild disagree. 1) externalizing the plan and letting the user audit/edit the plan while its working is "tool use", yes, but a very specialcase kind of tool use that, for example, operator and deep research use Temporal for. ofc we also saw this with Devin/Manus and i kinda think they're better 2) there is a form of primitive tree search that people are doing where they can spam out several different paths and run it a few steps ahead to gain information about optimal planning. You will see this with morph's launch at AIE. 3) plan meta reflection and reuse - again a form of tool use, but the devin and allhands folks have worked on this a lot more than most. my criticism of many agent definitions is that they generally do not take memory, planning, and auth seriously enough, and i think those 3 areas are my current bets for "alpha" in 2025. > I haven't seen a definition that specifically encompasses both short- and long-term memory before. here - https://docs.mem0.ai/core-concepts/memory-types#short-term-memory https://docs.mem0.ai/core-concepts/memory-types#short-term-m... - https://x.com/swyx/status/1915128966203236571 https://x.com/swyx/status/1915128966203236571
- Tokumei-no-hito 1y ago
  what is morphs launch at AIE?
  - deleted 1y ago
    [deleted]
  - deleted 1y ago
    [deleted]
  - nkko 1y ago
    Morph.so is an ai sandbox thing that has branching from state, prob doing a talk in SF on this at AI Engineer conf
    - Tokumei-no-hito 1y ago
      thanks
tough 1y ago
> It wasn't clear to me what the difference between "tool using" and "action execution" was. doing a lot of inference here, but could be a separation between -read- tool kinda actions, and -write/execute- (like running code/sending an email, etc) a bit weird from a coding perspective but idk
sebastiennight 1y ago
> similar to how ChatGPT's memory feature worked up until about four weeks ago What happened four weeks ago?
- swyx 1y ago
  they updated it opaquely - we no longer know anything about how it works https://www.latent.space/p/clippy-v-anton https://www.latent.space/p/clippy-v-anton
  - wunderwuzzi23 1y ago
    See my response to Simon above on some insights on how it works - I'll write up in detail in a blog post also when I get to it.
- simonw 1y ago
  They added a feature where it can reference content from any conversation you've had with it in the past. It's not clear how this works - probably some kind of vector search? Prior to that change "memory" was a tool call: https://simonwillison.net/2024/Oct/15/chatgpt-horoscopes/#how-memory-works https://simonwillison.net/2024/Oct/15/chatgpt-horoscopes/#ho...
  - wunderwuzzi23 1y ago
    Here is what I have been able to reverse engineer for o3... At high level it maintains about ~40 conversations in system prompt under a section called "recent conversation content". It only contains what the user typed, not assistant responses (probably due to prompt injection) - there a few corner cases though. :) There are other sections in the system prompt now that contain aggregated info, so recent conversations turn into user insights over time I believe. It can't actually "search" history afaik - that part I'm still wondering, as it was my first thought on how it might work... I also found a way to exfiltrate the recent content - so hopefully that will be fixed soon... Overall, this feature creates a lot of confusion and response quality declines at times too - and anything someone posts now online (like weird behavior or hallucinations,...) is likely influenced by their past conversations! So it will make it more difficult to understand what's really happening. I think it would be cool if "projects" would be entirely isolated with their own memories and history etc. or have different "profiles"
jsemrau 1y ago
Accurate memory access across memory types is still not really solved. That is the issue. Most agent frameworks from the main model providers are still quite spotty. Image of a table outlining where the major frameworks are: https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76107516-eba3-4a17-ba85-5ccd76180d29_960x540.png https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_... Here is also an article I wrote last year on the different types of memory: https://open.substack.com/pub/jdsemrau/p/memory-and-knowledge-for-agents?r=7fx0t&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true https://open.substack.com/pub/jdsemrau/p/memory-and-knowledg...

anonymousDan 1y ago

The hype around agent protocols reminds me of the emperor's new clothes. There's just nothing to it from a technical perspective.

esafak 1y ago
It's just something that needs to be done if you want LLMs to be useful. Everything does not have to be a technological marvel.
hendler 1y ago
HTML was XML for the web. Nothing to it from a technical perspective.
- pyuser583 1y ago
  XHTML was supposed to be revolutionary.
- padolsey 1y ago
  HTML was designed prior to XML fwiw, spinning off from SGML.
01100011 1y ago
Is it something like this: Ages and ages ago I was an EE turned programmer and everyone was hyping JUnit at the time. I had a customer ask for it on a project so fine I'll learn it. I kept thinking it was stupid because in my mind it barely did anything. But then I got it: it did barely do anything, but it did things you'd commonly need to do for testing and it did them in a somewhat standardized way that kept you from having to roll your own every time. Suddenly it didn't feel so stupid.
crystal_revenge 1y ago
Even worse is that virtually no one in this space even recognizes the fairly rich research into “agents” throughout the history of computer science. Mention John Holland’s work in adaptive systems, Hewitt’s actor model or even Minsky’s “Society of the Mind” and you’ll be met with blank stares. I do believe LLMs have the potential to make these older ideas relevant again and potentially create something amazing, but sadly the ignorant hype makes it virtually impossible to have intelligent conversations about these possibilities.

revskill 1y ago

Is AI Agent just LLM wrapper ? Is there anything more interesting to it ?

mindcrime 1y ago
I hate to say "it depends" but it, aaah, kinda depends. Nailing down a good definition of "Agent" has a been a problem dating back at least into the 1990's if not the 1980's. So, depending on which definition of "AI Agent" you're using, you arguably don't even need any LLM at all. Heck, using the most expansive definition I've seen, a mechanical thermostat counts. I don't know that I'd go that far, but I'll definitely say that I do not consider Agents to require use of LLM's. That said, the "Agent pattern du jour" is heavily based on using LLM's to provide the "brain" of the Agent and then Tool Calling to let it do things an LLM can't normally do. But still... depending on just what you do with those tool calls and any other code that sits in your Agent implementation then it certainly could be more than "just" an LLM wrapper. Nothing stops you from, for example, using the BDI architecture, implementing multi-level memory that's analogous to the way human memory works, wiring in some inductive learning, and throwing in some case-based reasoning, and an ontology based reasoning engine. Most people today aren't doing this, because they're mostly johnny-come-lately's that don't know anything about AI besides what they see on Twitter, Reddit, and LinkedIn; and wouldn't know BDI from BDSM. [1]: https://en.wikipedia.org/wiki/Belief%E2%80%93desire%E2%80%93intention_software_model https://en.wikipedia.org/wiki/Belief%E2%80%93desire%E2%80%93...
- freeamz 1y ago
  Agent seems like a process/worker/thread that is running LLM inference?
namaria 1y ago
Yes. Check the codebases. It's all prompting scaffolding. All of it. Chain of though, agents, tool using. It's just parsing the user inputs, adding text around it. "You are an expert X". That's the whole edifice.
MichaelMoser123 1y ago

sgt101 1y ago

Doesn't even mention KQML or FIPA - staggeringly ignorant.

eMPee584 1y ago
any experience with it? illuminate us lot if you will..
- sgt101 1y ago
  A very old report about FIPA that was rejected from AAMAS. https://github.com/sgt101/FIPA-Mirror-/blob/main/Experience_report_paper.pdf https://github.com/sgt101/FIPA-Mirror-/blob/main/Experience_...
ursaguild 1y ago
I was under the impression ACL supersedes KQML and was proposed by FIPA?
- sgt101 1y ago
  FIPA came after KQML and tried to learn from it and do something better. There were people who didn't think that it had succeeded in that. So it's debatable I think.
dang 1y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "When disagreeing, please reply to the argument instead of calling names. 'That is idiotic; 1 + 1 is 2, not 3' can be shortened to '1 + 1 is 2, not 3." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you know more than others, that's great, but in that case please share some of what you know, so the rest of us can learn. Putdowns don't help. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

arresin 1y ago

This is a very weird paper. They go through all these protocols yet haven’t don’t mention the most promising one (smolagents): https://github.com/huggingface/smolagents https://github.com/huggingface/smolagents

ursaguild 1y ago
smolagents by huggingface would be more of an agent framework. If it was discussed we would see smolagents/llamaindex/pydantic/etc with frameworks on figure 2. Several frameworks were left off in this paper as it focuses more on the protocols. https://huggingface.co/docs/smolagents/ https://huggingface.co/docs/smolagents/
- arresin 1y ago
  Good point. Appreciate the correction.

klabb3 1y ago

It’s good to see a new fresh push for protocols and interop. However, I don’t think it will do the thing we hope for, for non-technical reasons. During Web 2.0, we saw similar enthusiasm. Instead of AI agents or blockchain, every modern company had an API exposed. For instance, Gmail- and Facebook chat was usable with 3p client apps. What killed this was not tech, but business. The product wasn’t say social media, it was ad delivery. And using APIs was considered a bypass of funnels that they want to control. Today, if you go to a consumer service website, you will generally be met with a login/app wall. Even companies that charge money directly (say 23&me ad an egregious example) are also data hoarders. Apple is probably a better example. There’s no escape. The point is, protocols is the easy part. If the economics and incentives are the same as yesterday, we will see similar outcomes. Today, the consumer web is adversarial between provider ”platforms”, ad delivery, content creators, and the products themselves (ie the people who use them).

lorepieri 1y ago
I really like this analysis. But what about companies allowing agents to interact natively (via API or similar) getting more of the agents inbound since agents are more optimised to go there? If people want to use agents it will cause a lot of lost revenue for companies not allowing agents to interact natively.
- klabb3 1y ago
  It’s only lost revenue if the interactions were providing revenue in the first place. And for consumer tech, if you take away the ability to control the ad-funnels and engagement knobs, there is no revenue to gain. Why do you think all APIs got locked down? Nowadays even pure html delivery is getting locked down to prevent scraping. You think the SEO optimized recipe click holes are incentivized to deliver the content without the ads? I mean, it’s not impossible that consumers will ignore sites and services that don’t play with their AI. But even so, the content providers would need to find alternative revenue streams – otherwise they’d be bleeding money. My prediction is that tech companies will continue to compete for dominance over the entry points, apps and human interactions. It will be an adversarial space where coalitions of vertically integrated walled gardens can work. Basically how it is already.