4 ms·

P.S. nobody asked but since you made it this far - the next big problem in this space is fast becoming, what else do we need to be able to build these "headless

P.S. nobody asked but since you made it this far - the next big problem in this space is fast becoming, what else do we need to be able to build these "headless" or "outer loop" AI agents? Most frameworks do a bad job of handling any tool call that would be asynchronous or long running (imagine an agent calling a tool and having to hang for hours or days while waiting for a response from a human). Rewiring existing frameworks to support this is either hard or impossible, because you have to 1. fire the async request, 2. store the current context window somewhere, 3. catch a webhook, 4. map it back to the original agent/context, 5. append the webhook response to the context window, 6. resume execution with the updated context window. I have some ideas but I'll save that one for another post :) Thanks again for reading!

ehsanu1 2y ago
Temporal makes this easy and works great for such use cases. It's what I'm using for my own AI agents.
- dhorthy 2y ago
  ah very cool! are there any things you wish it did or any friction points? What are the things that "just work"?
  - ehsanu1 2y ago
    Essentially, you don't need to think about time and space. You just write more or less normal looking code, using the Temporal SDK. Except it actually can resume from arbitrarily long pauses, waiting as long as it needs to for some signal, without any special effort beyond using the SDK. You also automatically get great observability into all running workflows, seeing inputs and outputs at each step, etc. The cost of this is that you have to be careful in creating new versions of the workflow that are backwards compatible, and it's hard to understand backcompat requirements and easy to mess up. And, there's also additional infra you need, to run the Temporal server. Temporal Cloud isn't cheap at scale but does reduce that burden.
    - dhorthy 2y ago
      helpful - thanks! I have played with temporal a bit but have this thought that since most AI tools represent state as just a rolling context window, maybe you don't have to serialize and entire call stack and you can cut a bunch of corners. but we're all probably better off not investing that wheel
      - ehsanu1 2y ago
        IMO just a rolling message history works for only the simplest of AI tools. Useful agents will tend towards much more complex state that extends into specific verticals/domains.
        dhorthy 2y ago
        is that because of more deterministic AI flows like llm-as-judge, rag reranker, post-eval, etc? do you think something like langgraph state is sufficient?
lunarcave 2y ago
The MCP[1] that was announced by Anthropic has a solution to this problem, and it's pretty good at handling this use case. I've also been working on a solution to this problem via long-polling tools. [1] https://github.com/modelcontextprotocol https://github.com/modelcontextprotocol
- dhorthy 2y ago
  thanks for bringing this up. I just spent 2 hours last night digging into MCP - I'd love to learn more about how you think this solves the HitL problem. From my perspective MCP is more of a protocol for tool calling over the stdio wire, and the only situation it provides HitL is when human is sitting in the desktop app observing the agent synchronously? Again, genuinely looking to learn - where does MCP fit in for async/headless/ambient agents, beyond a solid protocol for remote tool calls?
  - potatoman22 2y ago
    You could implement some blocking HitL service/tool as an MCP server.
    - dhorthy 2y ago
      ah okay - I guess in that case, I would like chain a HitL step as an MCP server that wraps/chains to another tool that depends on approval? or is there a cleaner way to do that?
      - a1j9o94 2y ago
        Yeah, exactly. You would define a HitL server and the actions it implements would be API calls to your system.
        dhorthy 2y ago
        this is interesting. I will have to think more about how humanlayer can support an MCP integration/wrapper, it's not immediately obvious to me i do think that AI-calling-tools is insufficient to provide bidirectional communication rails for user input/review though...not disagreeing just maybe thinking out loud a little here
soheil 2y ago
Just to frame the problem slightly differently, if you had unlimited number of humans who could perform tasks as quickly as a computer this wouldn't be a problem need solving. So since we know that's the end state for any human-in-the-loop system then maybe it's worth solving that problem instead. A few things come to mind, divide the problem into chunks that can be solved in parallel by many people. Crowd source your platform so there are always people available with a very high SLA, just like cloud servers are today.
deleted 2y ago
[deleted]
chuckbear 2y ago
I've been using the DBOS (dbos.dev) framework to do this in a non-AI app that sends out render jobs, but imagine the process is no different for any other long-running request/response scenario: 1. Run the whole process in a workflow function. Give the run a unique ID, which can be used to automatically deduplicate runs, and will be used to look up the workflow later. This fires off the request in a "step" function, and calls `recv` to wait for a response. The request should include a key that can be used to calculate the workflow ID. 2. DBOS automatically "stores the context somewhere" because of the `recv` 3. A separate HTTP handler in DBOS catches the webhook, and uses the key in the response to calculate the ID of the workflow from #1. 4. The HTTP handler calls `send` with that ID, so that the workflow can pick up whatever response is sent 5. The original workflow resumes with the response from `send` 6. The original workflow can do whatever it wants with the response DBOS provides reliability around the whole thing (restarts any workflows in the case of any server restarts), and provides some tracing for the process out of the box, so it was quite simple to get started, and have it hosted in DBOS cloud , which also provides a public IP so that the external service can send a webhook response.

ehsanu1 2y ago

Temporal makes this easy and works great for such use cases. It's what I'm using for my own AI agents.

dhorthy 2y ago
ah very cool! are there any things you wish it did or any friction points? What are the things that "just work"?
- ehsanu1 2y ago
  Essentially, you don't need to think about time and space. You just write more or less normal looking code, using the Temporal SDK. Except it actually can resume from arbitrarily long pauses, waiting as long as it needs to for some signal, without any special effort beyond using the SDK. You also automatically get great observability into all running workflows, seeing inputs and outputs at each step, etc. The cost of this is that you have to be careful in creating new versions of the workflow that are backwards compatible, and it's hard to understand backcompat requirements and easy to mess up. And, there's also additional infra you need, to run the Temporal server. Temporal Cloud isn't cheap at scale but does reduce that burden.
  - dhorthy 2y ago
    helpful - thanks! I have played with temporal a bit but have this thought that since most AI tools represent state as just a rolling context window, maybe you don't have to serialize and entire call stack and you can cut a bunch of corners. but we're all probably better off not investing that wheel
    - ehsanu1 2y ago
      IMO just a rolling message history works for only the simplest of AI tools. Useful agents will tend towards much more complex state that extends into specific verticals/domains.
      - dhorthy 2y ago
        is that because of more deterministic AI flows like llm-as-judge, rag reranker, post-eval, etc? do you think something like langgraph state is sufficient?

lunarcave 2y ago

The MCP[1] that was announced by Anthropic has a solution to this problem, and it's pretty good at handling this use case. I've also been working on a solution to this problem via long-polling tools. [1] https://github.com/modelcontextprotocol https://github.com/modelcontextprotocol

dhorthy 2y ago
thanks for bringing this up. I just spent 2 hours last night digging into MCP - I'd love to learn more about how you think this solves the HitL problem. From my perspective MCP is more of a protocol for tool calling over the stdio wire, and the only situation it provides HitL is when human is sitting in the desktop app observing the agent synchronously? Again, genuinely looking to learn - where does MCP fit in for async/headless/ambient agents, beyond a solid protocol for remote tool calls?
- potatoman22 2y ago
  You could implement some blocking HitL service/tool as an MCP server.
  - dhorthy 2y ago
    ah okay - I guess in that case, I would like chain a HitL step as an MCP server that wraps/chains to another tool that depends on approval? or is there a cleaner way to do that?
    - a1j9o94 2y ago
      Yeah, exactly. You would define a HitL server and the actions it implements would be API calls to your system.
      - dhorthy 2y ago
        this is interesting. I will have to think more about how humanlayer can support an MCP integration/wrapper, it's not immediately obvious to me i do think that AI-calling-tools is insufficient to provide bidirectional communication rails for user input/review though...not disagreeing just maybe thinking out loud a little here

soheil 2y ago

Just to frame the problem slightly differently, if you had unlimited number of humans who could perform tasks as quickly as a computer this wouldn't be a problem need solving. So since we know that's the end state for any human-in-the-loop system then maybe it's worth solving that problem instead. A few things come to mind, divide the problem into chunks that can be solved in parallel by many people. Crowd source your platform so there are always people available with a very high SLA, just like cloud servers are today.

deleted 2y ago

[deleted]

chuckbear 2y ago

I've been using the DBOS (dbos.dev) framework to do this in a non-AI app that sends out render jobs, but imagine the process is no different for any other long-running request/response scenario: 1. Run the whole process in a workflow function. Give the run a unique ID, which can be used to automatically deduplicate runs, and will be used to look up the workflow later. This fires off the request in a "step" function, and calls `recv` to wait for a response. The request should include a key that can be used to calculate the workflow ID. 2. DBOS automatically "stores the context somewhere" because of the `recv` 3. A separate HTTP handler in DBOS catches the webhook, and uses the key in the response to calculate the ID of the workflow from #1. 4. The HTTP handler calls `send` with that ID, so that the workflow can pick up whatever response is sent 5. The original workflow resumes with the response from `send` 6. The original workflow can do whatever it wants with the response DBOS provides reliability around the whole thing (restarts any workflows in the case of any server restarts), and provides some tracing for the process out of the box, so it was quite simple to get started, and have it hosted in DBOS cloud , which also provides a public IP so that the external service can send a webhook response.