4 ms·

The more I read about ML, the more I begin to believe that - psychologically speaking - hierarchy (esp. graph structures, trees) are absolutely core to advanced

The more I read about ML, the more I begin to believe that - psychologically speaking - hierarchy (esp. graph structures, trees) are absolutely core to advanced information processing in general.

blurbleblurble 3y ago
This article is about how non-hierarchical graphs, those with cycles, are performing better than trees or chains.
- theptip 3y ago
  I suspect that you'll still find strong hierarchy in an optimized/well-performing graph of thought. The human brain, for example, also has recurrence, but it's limited. It seems pretty intuitive that you'd get a "task / subtask" split for example, with feedback from the latter, but semantic content largely flowing from the former to the latter.
brutusborn 3y ago
I won’t pretend to understand it, but this reminds me of the idea of markov blankets when using the free energy principle to model congition. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7955287/#:~:text=The%20free%2Denergy%20principle%20begins,state%20(Friston%2C%202020) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7955287/#:~:tex....
visarga 3y ago
We got symbolic AI sneaked into the connectionist model by making a graph of thoughts. A graph can explicitly implement any algorithm or data structure. They could make it more efficient by implementing a kind of "hard attention". Each token should have access to a sparse subset of the whole input, so it would be like a node in a graph only having access to a few neighbours. Could solve the very large context issue. This can also be parallelised, running all thought nodes in parallel, of course each with a sparse view of the whole input making it much faster. For example when reading a long book, the model would spawn nodes for each person, location or event of interest, and they would track the source text as the action develops. A mental map of the book. That would surely help a model deal with many moving pieces of information. Or when solving a problem, the model could spawn a node to work on a subproblem, parametrised by the parent node with the right inputs. Then the node would report back with the answer and the parent continues. This would work with recursive calls. The new cpu is the LLM and the clock tick is 1 token.
- barrenko 3y ago
  Could you expand on "A graph can explicitly implement any algorithm or data structure."?
  - visarga 3y ago
    You could create a node for each execution step, or data field.
    - Vox_Leone 3y ago
      And thus you could use UML to formalize prompting. An activity diagram can be viewed as a chain of thought. Fulfilling the UML promise.
  - gryn 3y ago
    you can use graph transformation to perform general computation. https://en.wikipedia.org/wiki/Graph_rewriting https://en.wikipedia.org/wiki/Graph_rewriting probably not was GP meant, but something along those lines.
    - barrenko 3y ago
      Appreciate it.

The more I read about ML, the more I begin to believe that - psychologically speaking - hierarchy (esp. graph structures, trees) are absolutely core to advanced information processing in general.

blurbleblurble 3y ago

This article is about how non-hierarchical graphs, those with cycles, are performing better than trees or chains.

theptip 3y ago
I suspect that you'll still find strong hierarchy in an optimized/well-performing graph of thought. The human brain, for example, also has recurrence, but it's limited. It seems pretty intuitive that you'd get a "task / subtask" split for example, with feedback from the latter, but semantic content largely flowing from the former to the latter.

brutusborn 3y ago

I won’t pretend to understand it, but this reminds me of the idea of markov blankets when using the free energy principle to model congition. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7955287/#:~:text=The%20free%2Denergy%20principle%20begins,state%20(Friston%2C%202020) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7955287/#:~:tex....

visarga 3y ago

We got symbolic AI sneaked into the connectionist model by making a graph of thoughts. A graph can explicitly implement any algorithm or data structure. They could make it more efficient by implementing a kind of "hard attention". Each token should have access to a sparse subset of the whole input, so it would be like a node in a graph only having access to a few neighbours. Could solve the very large context issue. This can also be parallelised, running all thought nodes in parallel, of course each with a sparse view of the whole input making it much faster. For example when reading a long book, the model would spawn nodes for each person, location or event of interest, and they would track the source text as the action develops. A mental map of the book. That would surely help a model deal with many moving pieces of information. Or when solving a problem, the model could spawn a node to work on a subproblem, parametrised by the parent node with the right inputs. Then the node would report back with the answer and the parent continues. This would work with recursive calls. The new cpu is the LLM and the clock tick is 1 token.

barrenko 3y ago
Could you expand on "A graph can explicitly implement any algorithm or data structure."?
- visarga 3y ago
  You could create a node for each execution step, or data field.
  - Vox_Leone 3y ago
    And thus you could use UML to formalize prompting. An activity diagram can be viewed as a chain of thought. Fulfilling the UML promise.
- gryn 3y ago
  you can use graph transformation to perform general computation. https://en.wikipedia.org/wiki/Graph_rewriting https://en.wikipedia.org/wiki/Graph_rewriting probably not was GP meant, but something along those lines.
  - barrenko 3y ago
    Appreciate it.