13 ms·

It is hard to put a pin on this one because there are so many thing wrong with this definition. There are agent frameworks that are not rebranded workflow tools

by _pdp_ 2y ago

It is hard to put a pin on this one because there are so many thing wrong with this definition. There are agent frameworks that are not rebranded workflow tools too. I don't think this article helps explain anything except putting the intended audience in the same box of mind we were stuck since the invention of programming - i.e. it does not help. Forget about boxes and deterministic control and start thinking of error tolerance and recovery. That is what agents are all about.

zh2408 2y ago
Hey, sorry for the confusion. This tutorial is focusing on the low-level internals of how agents are implemented—much like how intelligent large language models still boil down to matrix multiplications at their core.
- adamnemecek 2y ago
  Despite the memes, this reductivism is not exactly insightful. Like why stop there? Matrix multiplication is just a bunch of dot product. Which in turn is just cos and magnitude. What insights were generated from this?
  - ethanwillis 2y ago
    The reductionism is insightful when it comes to providing an implementation with those specific details in mind. In the case of LLMs knowing it does boil down to matrix multiplication is insightful and useful because now you know what kind of hardware is best suited to executing a model. What is actually not insightful or useful is believing LLMs are AGI or conscious.
    - ForTheKidz 2y ago
      Belief is generally not insightful or useful by definition. Then again, I don't think anyone who can follow this article believed that LLMs were conscious to begin with, so I'm not sure what your point is. You're preaching on behalf of a demographic that won't read this article to begin with, and presumably the people who are can see how useless, distracting, and unproductive this reductionism is.
      - ethanwillis 2y ago
        The person I'm replying to literally has a startup targeted at making AGI coming from the LLM hype cycle.
        adamnemecek 2y ago
        > coming from the LLM hype cycle Our approach is so unrelated to any of the other hyped up stuff. We have not written a single line of ML, it has been all math & physics until now.
        ForTheKidz 2y ago
        I believe this was precluded by the hedging of people who could follow the article. I have a difficult time imagining a person who can both understand how current LLMs work and still buy into Kurzweil. Pursue the hypothesis? Sure. But belief is a different beast entirely. It's not even clear AGI is a meaningful concept yet, and I'd bet my life savings everyone reading this comment in 2025 will die before it's answered. Skepticism is the barometer.
  - zh2408 2y ago
    It’s kind of like the different levels of abstraction. For example, for software projects, the algorithmic level is where most people focus because that’s typically where the biggest optimizations happen. But in some critical scenarios, you have to peel back those layers—down to how the hardware or compiler works—to make the best choices (like picking the right CPU/GPU). Likewise, with agents, you can work with high-level abstractions for most applications. But if you need to optimize or compare different approaches (tool use vs. MCP vs. prompt-based, for instance), you have to dig deeper into how they’re actually implemented.
  - _factor 2y ago
    If you can reduce complex matrix multiplications into simpler terms, then you may be able to focus the training based on those constraints to increase performance/efficiency.
  - xg15 2y ago
    ok, then how would you do it?
  - godelski 2y ago
    > this reductivism is not exactly insightful. I really agree with this. I think it has been bad for a lot of people's understanding when they have trivialized ML to "just matrix multiplications" (or GMMs). This does not help differentiate AI/ML from... well.. really any data processing algorithm. Matrices are fairly general structures in mathematics and you can formulate almost anything as one. In fact, this is a very common way to parallelize or speed up programs (e.g. numpy vectorization). We wouldn't call least squares, even a bunch of them, ML nor would we call rasterization or ray tracing. Fundamentally all these things are "just GMMs". It also does not make apparent any differentiation from important distinctions like Linear Networks, CNNs, or Transformers. It brushes off a key element, the activation function, which is necessary for neural nets to do non-linear transformations! And what about the residual units? These are one of the most important factors in enabling Deep Learning. They're "just" addition. So we say it's all just matrix addition since we can convert multiplication to addition? There is such a thing as oversimplification and I worry that we have hyper-optimized (over-optimized) for this. So I agree, saying they just "boil down to matrix multiplications" is fundamentally misleading. It provides no insight and only serves to mislead people.
- godelski 2y ago
  > This tutorial is focusing on the low-level internals of how agents are implemented We have very different definitions of what "low-level" means. Exact opposites in fact. "Low-level" means in the inner workings. Like a low-level language is assembly (some consider C low-level but this is debatable), whereas Python would be high-level. I don't think this tutorial is "near the metal" of LLMs nor do I think it should be considering it is aimed at "Dummies". Low-level would really need to get into the inner workings of the processing, probing agents, and getting into the weeds.
  - zh2408 2y ago
    By low-level, it is with respect to the agent interface. The original purpose is to help people understand how the inner agent framework is internally implemented, like those: OpenAI Agents: https://github.com/openai/openai-agents-python/blob/48ff99bb736249e99251eb2c7ecf00237488c17a/src/agents/run.py#L119 https://github.com/openai/openai-agents-python/blob/48ff99bb... Pydantic Agents: https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299382c22a8e3372638885e18286/pydantic_ai_slim/pydantic_ai/_agent_graph.py#L779 https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299... Langchain: https://github.com/langchain-ai/langchain/blob/4d1d726e61ed58b39278903262d19bbe9f010772/libs/langchain/langchain/agents/agent_iterator.py#L174 https://github.com/langchain-ai/langchain/blob/4d1d726e61ed5... LangGraph: https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2c19b634ed7af0d551ad327e25d7/libs/cli/examples/graphs/agent.py#L56 https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2...
  - windsignaling 2y ago
    I think "low-level" is relative to what's being discussed. Low-level for LLMs would have to do with how transformer layers are implemented (self-attention layer, layer norms, etc.) whereas low-level for agents would be the graph structure. Although I personally don't think the graph implementation for agents is necessarily as established or widely standardized, it's helpful to know about why such an implementation was chosen and how it works. > the inner workings of the processing, probing agents, and getting into the weeds These feel to me like empty words... "inner workings of the processing"? You can say that about anything.
xg15 2y ago
what exactly do you mean with "error tolerance and recovery"?
heyitsguay 2y ago
Could you give some examples of the agent frameworks you're referring to? I'd love to see some examples that go beyond the graph pattern! Thank you
- _pdp_ 2y ago
  The agentic ai capabilities of chatbotkit.com has nothing to do with workflows. The graph rendering is simply for illustrative purposes and most to cater for people who think in terms of graphs but the underlaying mechanics are not nodes and edges and a flow that goes from one to the next.
vincston 2y ago
Why forget about boxes and deterministic control and start thinking of error tolerance and recovery? I know, that LLMs are statistical models, but can you not use patterns to enforce a deterministic outcome? (Single responsibility for each agent, retrying llm calls, rephrasing prompts, etc?)
ForTheKidz 2y ago
> There are agent frameworks that are not rebranded workflow tools too. To me "workflow" is just what agent means: the rules under which an automated action occurs. Without some central concept "agent" just a magic wand that does stuff that may or may not be what you want it to do. If we can't use state machines at all I'm just going to go out and say LLMs are a dead end. State machines are the bread and butter of reliable software. > Forget about boxes and deterministic control and start thinking of error tolerance and recovery. First you'd have to define what an error even is. Then you're just writing deterministic software again (a workflow), just with less confidence. Nice for stuff with low risk and confidence to begin with (eg semantic analysis etc whose error tends to wash out in aggregate), but not for stuff acting on my behalf. LLMs are cool bits of software, but I can't say I see much use for "agents" whose behavior is not well-defined and whose non-determinism is formally bounded.
- infecto 2y ago
  It’s getting pedantic, but the key idea is that Agents can solve problems traditional state machine-based workflows couldn't. Your point is moot since many of these modern workflows already use LLMs as gating functions to determine the next steps. It’s a different way of approaching problems, and while the future is uncertain, LLMs have moved beyond being just "cool software" to becoming genuinely useful in specific domains.
  - ForTheKidz 2y ago
    Hmm, maybe you are referring to something specific with "workflow". I'm envision a visual graph with a ui for each node and connection, or maybe a makefile on the other end of the spectrum. What are you envisioning? Anyway, LLMs will remain at "cool software" like other niche-specific patterns until I see something general emerge. You'd have to pitch LLMs pretty savvily to show it as a clear value-add. Engineers are extremely expensive, so LLMs need to have a very low error rate to be integrated into the revenue-path of a product to not incur higher costs or a lower-quality service. I still see text- and code-generation for immediate consumption by a human (or possible classification to be reviewed by a human) as the only viable uses cases today. It's just way too easy to manipulate them with standard english.

by _pdp_ 2y ago

zh2408 2y ago

Hey, sorry for the confusion. This tutorial is focusing on the low-level internals of how agents are implemented—much like how intelligent large language models still boil down to matrix multiplications at their core.

adamnemecek 2y ago
Despite the memes, this reductivism is not exactly insightful. Like why stop there? Matrix multiplication is just a bunch of dot product. Which in turn is just cos and magnitude. What insights were generated from this?
- ethanwillis 2y ago
  The reductionism is insightful when it comes to providing an implementation with those specific details in mind. In the case of LLMs knowing it does boil down to matrix multiplication is insightful and useful because now you know what kind of hardware is best suited to executing a model. What is actually not insightful or useful is believing LLMs are AGI or conscious.
  - ForTheKidz 2y ago
    Belief is generally not insightful or useful by definition. Then again, I don't think anyone who can follow this article believed that LLMs were conscious to begin with, so I'm not sure what your point is. You're preaching on behalf of a demographic that won't read this article to begin with, and presumably the people who are can see how useless, distracting, and unproductive this reductionism is.
    - ethanwillis 2y ago
      The person I'm replying to literally has a startup targeted at making AGI coming from the LLM hype cycle.
      - adamnemecek 2y ago
        > coming from the LLM hype cycle Our approach is so unrelated to any of the other hyped up stuff. We have not written a single line of ML, it has been all math & physics until now.
      - ForTheKidz 2y ago
        I believe this was precluded by the hedging of people who could follow the article. I have a difficult time imagining a person who can both understand how current LLMs work and still buy into Kurzweil. Pursue the hypothesis? Sure. But belief is a different beast entirely. It's not even clear AGI is a meaningful concept yet, and I'd bet my life savings everyone reading this comment in 2025 will die before it's answered. Skepticism is the barometer.
- zh2408 2y ago
  It’s kind of like the different levels of abstraction. For example, for software projects, the algorithmic level is where most people focus because that’s typically where the biggest optimizations happen. But in some critical scenarios, you have to peel back those layers—down to how the hardware or compiler works—to make the best choices (like picking the right CPU/GPU). Likewise, with agents, you can work with high-level abstractions for most applications. But if you need to optimize or compare different approaches (tool use vs. MCP vs. prompt-based, for instance), you have to dig deeper into how they’re actually implemented.
- _factor 2y ago
  If you can reduce complex matrix multiplications into simpler terms, then you may be able to focus the training based on those constraints to increase performance/efficiency.
- xg15 2y ago
  ok, then how would you do it?
- godelski 2y ago
  > this reductivism is not exactly insightful. I really agree with this. I think it has been bad for a lot of people's understanding when they have trivialized ML to "just matrix multiplications" (or GMMs). This does not help differentiate AI/ML from... well.. really any data processing algorithm. Matrices are fairly general structures in mathematics and you can formulate almost anything as one. In fact, this is a very common way to parallelize or speed up programs (e.g. numpy vectorization). We wouldn't call least squares, even a bunch of them, ML nor would we call rasterization or ray tracing. Fundamentally all these things are "just GMMs". It also does not make apparent any differentiation from important distinctions like Linear Networks, CNNs, or Transformers. It brushes off a key element, the activation function, which is necessary for neural nets to do non-linear transformations! And what about the residual units? These are one of the most important factors in enabling Deep Learning. They're "just" addition. So we say it's all just matrix addition since we can convert multiplication to addition? There is such a thing as oversimplification and I worry that we have hyper-optimized (over-optimized) for this. So I agree, saying they just "boil down to matrix multiplications" is fundamentally misleading. It provides no insight and only serves to mislead people.
godelski 2y ago
> This tutorial is focusing on the low-level internals of how agents are implemented We have very different definitions of what "low-level" means. Exact opposites in fact. "Low-level" means in the inner workings. Like a low-level language is assembly (some consider C low-level but this is debatable), whereas Python would be high-level. I don't think this tutorial is "near the metal" of LLMs nor do I think it should be considering it is aimed at "Dummies". Low-level would really need to get into the inner workings of the processing, probing agents, and getting into the weeds.
- zh2408 2y ago
  By low-level, it is with respect to the agent interface. The original purpose is to help people understand how the inner agent framework is internally implemented, like those: OpenAI Agents: https://github.com/openai/openai-agents-python/blob/48ff99bb736249e99251eb2c7ecf00237488c17a/src/agents/run.py#L119 https://github.com/openai/openai-agents-python/blob/48ff99bb... Pydantic Agents: https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299382c22a8e3372638885e18286/pydantic_ai_slim/pydantic_ai/_agent_graph.py#L779 https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299... Langchain: https://github.com/langchain-ai/langchain/blob/4d1d726e61ed58b39278903262d19bbe9f010772/libs/langchain/langchain/agents/agent_iterator.py#L174 https://github.com/langchain-ai/langchain/blob/4d1d726e61ed5... LangGraph: https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2c19b634ed7af0d551ad327e25d7/libs/cli/examples/graphs/agent.py#L56 https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2...
- windsignaling 2y ago
  I think "low-level" is relative to what's being discussed. Low-level for LLMs would have to do with how transformer layers are implemented (self-attention layer, layer norms, etc.) whereas low-level for agents would be the graph structure. Although I personally don't think the graph implementation for agents is necessarily as established or widely standardized, it's helpful to know about why such an implementation was chosen and how it works. > the inner workings of the processing, probing agents, and getting into the weeds These feel to me like empty words... "inner workings of the processing"? You can say that about anything.

xg15 2y ago

what exactly do you mean with "error tolerance and recovery"?

heyitsguay 2y ago

Could you give some examples of the agent frameworks you're referring to? I'd love to see some examples that go beyond the graph pattern! Thank you

_pdp_ 2y ago
The agentic ai capabilities of chatbotkit.com has nothing to do with workflows. The graph rendering is simply for illustrative purposes and most to cater for people who think in terms of graphs but the underlaying mechanics are not nodes and edges and a flow that goes from one to the next.

vincston 2y ago

Why forget about boxes and deterministic control and start thinking of error tolerance and recovery? I know, that LLMs are statistical models, but can you not use patterns to enforce a deterministic outcome? (Single responsibility for each agent, retrying llm calls, rephrasing prompts, etc?)

ForTheKidz 2y ago

> There are agent frameworks that are not rebranded workflow tools too. To me "workflow" is just what agent means: the rules under which an automated action occurs. Without some central concept "agent" just a magic wand that does stuff that may or may not be what you want it to do. If we can't use state machines at all I'm just going to go out and say LLMs are a dead end. State machines are the bread and butter of reliable software. > Forget about boxes and deterministic control and start thinking of error tolerance and recovery. First you'd have to define what an error even is. Then you're just writing deterministic software again (a workflow), just with less confidence. Nice for stuff with low risk and confidence to begin with (eg semantic analysis etc whose error tends to wash out in aggregate), but not for stuff acting on my behalf. LLMs are cool bits of software, but I can't say I see much use for "agents" whose behavior is not well-defined and whose non-determinism is formally bounded.

infecto 2y ago
It’s getting pedantic, but the key idea is that Agents can solve problems traditional state machine-based workflows couldn't. Your point is moot since many of these modern workflows already use LLMs as gating functions to determine the next steps. It’s a different way of approaching problems, and while the future is uncertain, LLMs have moved beyond being just "cool software" to becoming genuinely useful in specific domains.
- ForTheKidz 2y ago
  Hmm, maybe you are referring to something specific with "workflow". I'm envision a visual graph with a ui for each node and connection, or maybe a makefile on the other end of the spectrum. What are you envisioning? Anyway, LLMs will remain at "cool software" like other niche-specific patterns until I see something general emerge. You'd have to pitch LLMs pretty savvily to show it as a clear value-add. Engineers are extremely expensive, so LLMs need to have a very low error rate to be integrated into the revenue-path of a product to not incur higher costs or a lower-quality service. I still see text- and code-generation for immediate consumption by a human (or possible classification to be reviewed by a human) as the only viable uses cases today. It's just way too easy to manipulate them with standard english.