4 ms·

Hey, sorry for the confusion. This tutorial is focusing on the low-level internals of how agents are implemented—much like how intelligent large language models

by zh2408 2y ago

Hey, sorry for the confusion. This tutorial is focusing on the low-level internals of how agents are implemented—much like how intelligent large language models still boil down to matrix multiplications at their core.

adamnemecek 2y ago
Despite the memes, this reductivism is not exactly insightful. Like why stop there? Matrix multiplication is just a bunch of dot product. Which in turn is just cos and magnitude. What insights were generated from this?
- ethanwillis 2y ago
  The reductionism is insightful when it comes to providing an implementation with those specific details in mind. In the case of LLMs knowing it does boil down to matrix multiplication is insightful and useful because now you know what kind of hardware is best suited to executing a model. What is actually not insightful or useful is believing LLMs are AGI or conscious.
  - ForTheKidz 2y ago
    Belief is generally not insightful or useful by definition. Then again, I don't think anyone who can follow this article believed that LLMs were conscious to begin with, so I'm not sure what your point is. You're preaching on behalf of a demographic that won't read this article to begin with, and presumably the people who are can see how useless, distracting, and unproductive this reductionism is.
    - ethanwillis 2y ago
      The person I'm replying to literally has a startup targeted at making AGI coming from the LLM hype cycle.
      - adamnemecek 2y ago
        > coming from the LLM hype cycle Our approach is so unrelated to any of the other hyped up stuff. We have not written a single line of ML, it has been all math & physics until now.
      - ForTheKidz 2y ago
        I believe this was precluded by the hedging of people who could follow the article. I have a difficult time imagining a person who can both understand how current LLMs work and still buy into Kurzweil. Pursue the hypothesis? Sure. But belief is a different beast entirely. It's not even clear AGI is a meaningful concept yet, and I'd bet my life savings everyone reading this comment in 2025 will die before it's answered. Skepticism is the barometer.
- zh2408 2y ago
  It’s kind of like the different levels of abstraction. For example, for software projects, the algorithmic level is where most people focus because that’s typically where the biggest optimizations happen. But in some critical scenarios, you have to peel back those layers—down to how the hardware or compiler works—to make the best choices (like picking the right CPU/GPU). Likewise, with agents, you can work with high-level abstractions for most applications. But if you need to optimize or compare different approaches (tool use vs. MCP vs. prompt-based, for instance), you have to dig deeper into how they’re actually implemented.
- _factor 2y ago
  If you can reduce complex matrix multiplications into simpler terms, then you may be able to focus the training based on those constraints to increase performance/efficiency.
- xg15 2y ago
  ok, then how would you do it?
- godelski 2y ago
  > this reductivism is not exactly insightful. I really agree with this. I think it has been bad for a lot of people's understanding when they have trivialized ML to "just matrix multiplications" (or GMMs). This does not help differentiate AI/ML from... well.. really any data processing algorithm. Matrices are fairly general structures in mathematics and you can formulate almost anything as one. In fact, this is a very common way to parallelize or speed up programs (e.g. numpy vectorization). We wouldn't call least squares, even a bunch of them, ML nor would we call rasterization or ray tracing. Fundamentally all these things are "just GMMs". It also does not make apparent any differentiation from important distinctions like Linear Networks, CNNs, or Transformers. It brushes off a key element, the activation function, which is necessary for neural nets to do non-linear transformations! And what about the residual units? These are one of the most important factors in enabling Deep Learning. They're "just" addition. So we say it's all just matrix addition since we can convert multiplication to addition? There is such a thing as oversimplification and I worry that we have hyper-optimized (over-optimized) for this. So I agree, saying they just "boil down to matrix multiplications" is fundamentally misleading. It provides no insight and only serves to mislead people.
godelski 2y ago
> This tutorial is focusing on the low-level internals of how agents are implemented We have very different definitions of what "low-level" means. Exact opposites in fact. "Low-level" means in the inner workings. Like a low-level language is assembly (some consider C low-level but this is debatable), whereas Python would be high-level. I don't think this tutorial is "near the metal" of LLMs nor do I think it should be considering it is aimed at "Dummies". Low-level would really need to get into the inner workings of the processing, probing agents, and getting into the weeds.
- zh2408 2y ago
  By low-level, it is with respect to the agent interface. The original purpose is to help people understand how the inner agent framework is internally implemented, like those: OpenAI Agents: https://github.com/openai/openai-agents-python/blob/48ff99bb736249e99251eb2c7ecf00237488c17a/src/agents/run.py#L119 https://github.com/openai/openai-agents-python/blob/48ff99bb... Pydantic Agents: https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299382c22a8e3372638885e18286/pydantic_ai_slim/pydantic_ai/_agent_graph.py#L779 https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299... Langchain: https://github.com/langchain-ai/langchain/blob/4d1d726e61ed58b39278903262d19bbe9f010772/libs/langchain/langchain/agents/agent_iterator.py#L174 https://github.com/langchain-ai/langchain/blob/4d1d726e61ed5... LangGraph: https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2c19b634ed7af0d551ad327e25d7/libs/cli/examples/graphs/agent.py#L56 https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2...
- windsignaling 2y ago
  I think "low-level" is relative to what's being discussed. Low-level for LLMs would have to do with how transformer layers are implemented (self-attention layer, layer norms, etc.) whereas low-level for agents would be the graph structure. Although I personally don't think the graph implementation for agents is necessarily as established or widely standardized, it's helpful to know about why such an implementation was chosen and how it works. > the inner workings of the processing, probing agents, and getting into the weeds These feel to me like empty words... "inner workings of the processing"? You can say that about anything.

by zh2408 2y ago

adamnemecek 2y ago

Despite the memes, this reductivism is not exactly insightful. Like why stop there? Matrix multiplication is just a bunch of dot product. Which in turn is just cos and magnitude. What insights were generated from this?

ethanwillis 2y ago
The reductionism is insightful when it comes to providing an implementation with those specific details in mind. In the case of LLMs knowing it does boil down to matrix multiplication is insightful and useful because now you know what kind of hardware is best suited to executing a model. What is actually not insightful or useful is believing LLMs are AGI or conscious.
- ForTheKidz 2y ago
  Belief is generally not insightful or useful by definition. Then again, I don't think anyone who can follow this article believed that LLMs were conscious to begin with, so I'm not sure what your point is. You're preaching on behalf of a demographic that won't read this article to begin with, and presumably the people who are can see how useless, distracting, and unproductive this reductionism is.
  - ethanwillis 2y ago
    The person I'm replying to literally has a startup targeted at making AGI coming from the LLM hype cycle.
    - adamnemecek 2y ago
      > coming from the LLM hype cycle Our approach is so unrelated to any of the other hyped up stuff. We have not written a single line of ML, it has been all math & physics until now.
    - ForTheKidz 2y ago
      I believe this was precluded by the hedging of people who could follow the article. I have a difficult time imagining a person who can both understand how current LLMs work and still buy into Kurzweil. Pursue the hypothesis? Sure. But belief is a different beast entirely. It's not even clear AGI is a meaningful concept yet, and I'd bet my life savings everyone reading this comment in 2025 will die before it's answered. Skepticism is the barometer.
zh2408 2y ago
It’s kind of like the different levels of abstraction. For example, for software projects, the algorithmic level is where most people focus because that’s typically where the biggest optimizations happen. But in some critical scenarios, you have to peel back those layers—down to how the hardware or compiler works—to make the best choices (like picking the right CPU/GPU). Likewise, with agents, you can work with high-level abstractions for most applications. But if you need to optimize or compare different approaches (tool use vs. MCP vs. prompt-based, for instance), you have to dig deeper into how they’re actually implemented.
_factor 2y ago
If you can reduce complex matrix multiplications into simpler terms, then you may be able to focus the training based on those constraints to increase performance/efficiency.
xg15 2y ago
ok, then how would you do it?
godelski 2y ago
> this reductivism is not exactly insightful. I really agree with this. I think it has been bad for a lot of people's understanding when they have trivialized ML to "just matrix multiplications" (or GMMs). This does not help differentiate AI/ML from... well.. really any data processing algorithm. Matrices are fairly general structures in mathematics and you can formulate almost anything as one. In fact, this is a very common way to parallelize or speed up programs (e.g. numpy vectorization). We wouldn't call least squares, even a bunch of them, ML nor would we call rasterization or ray tracing. Fundamentally all these things are "just GMMs". It also does not make apparent any differentiation from important distinctions like Linear Networks, CNNs, or Transformers. It brushes off a key element, the activation function, which is necessary for neural nets to do non-linear transformations! And what about the residual units? These are one of the most important factors in enabling Deep Learning. They're "just" addition. So we say it's all just matrix addition since we can convert multiplication to addition? There is such a thing as oversimplification and I worry that we have hyper-optimized (over-optimized) for this. So I agree, saying they just "boil down to matrix multiplications" is fundamentally misleading. It provides no insight and only serves to mislead people.

godelski 2y ago

> This tutorial is focusing on the low-level internals of how agents are implemented We have very different definitions of what "low-level" means. Exact opposites in fact. "Low-level" means in the inner workings. Like a low-level language is assembly (some consider C low-level but this is debatable), whereas Python would be high-level. I don't think this tutorial is "near the metal" of LLMs nor do I think it should be considering it is aimed at "Dummies". Low-level would really need to get into the inner workings of the processing, probing agents, and getting into the weeds.

zh2408 2y ago
By low-level, it is with respect to the agent interface. The original purpose is to help people understand how the inner agent framework is internally implemented, like those: OpenAI Agents: https://github.com/openai/openai-agents-python/blob/48ff99bb736249e99251eb2c7ecf00237488c17a/src/agents/run.py#L119 https://github.com/openai/openai-agents-python/blob/48ff99bb... Pydantic Agents: https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299382c22a8e3372638885e18286/pydantic_ai_slim/pydantic_ai/_agent_graph.py#L779 https://github.com/pydantic/pydantic-ai/blob/4c0f384a0626299... Langchain: https://github.com/langchain-ai/langchain/blob/4d1d726e61ed58b39278903262d19bbe9f010772/libs/langchain/langchain/agents/agent_iterator.py#L174 https://github.com/langchain-ai/langchain/blob/4d1d726e61ed5... LangGraph: https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2c19b634ed7af0d551ad327e25d7/libs/cli/examples/graphs/agent.py#L56 https://github.com/langchain-ai/langgraph/blob/24f7d7c4399e2...
windsignaling 2y ago
I think "low-level" is relative to what's being discussed. Low-level for LLMs would have to do with how transformer layers are implemented (self-attention layer, layer norms, etc.) whereas low-level for agents would be the graph structure. Although I personally don't think the graph implementation for agents is necessarily as established or widely standardized, it's helpful to know about why such an implementation was chosen and how it works. > the inner workings of the processing, probing agents, and getting into the weeds These feel to me like empty words... "inner workings of the processing"? You can say that about anything.