4 ms·

> One amazing realization one can get from this is, what is the conceptual equivalent of the transition functions that connect all different manifolds in this L

> One amazing realization one can get from this is, what is the conceptual equivalent of the transition functions that connect all different manifolds in this LLM space? Could you elaborate on what you mean by "transition functions" here?

evertedsphere 3y ago
likely https://en.m.wikipedia.org/wiki/Atlas_(topology)#Transition_maps https://en.m.wikipedia.org/wiki/Atlas_(topology)#Transition_... although the quoted sentence does not make sense to me; transition maps connect different patches of one manifold. it's possible the "LLM space" gp is talking about is a parameter space of some nature each of whose points is a manifold, but that seems like a stretch
- moralestapia 3y ago
  You're right, that's a bit ambiguous. Instead of "all different manifolds" I should have written "all different ways to define a manifold". Also, now that I think about it, it may not necessarily be all of them. The important thing here is that, however you define them and their transition maps, you'll find they're very much alike. As if there was some sort of general structure that is highly preferred over others ...
  - evertedsphere 3y ago
    "if you look at the (chart) transition maps for different atlases on this manifold, they tend to look similar for different atlases" what is the manifold here and what evidence do you have for this / what does it look like when you "define" them?
    - moralestapia 3y ago
      See, https://transformer-circuits.pub/2023/may-update/index.html#feature-manifolds https://transformer-circuits.pub/2023/may-update/index.html#.... They also reference, https://distill.pub/2020/circuits/equivariance/ https://distill.pub/2020/circuits/equivariance/. I'd also add, https://distill.pub/2021/multimodal-neurons/ https://distill.pub/2021/multimodal-neurons/.
- codethief 3y ago
  Yeah, that's the definition I was thinking of, too, but the sentence didn't make sense to me, either.

evertedsphere 3y ago

likely https://en.m.wikipedia.org/wiki/Atlas_(topology)#Transition_maps https://en.m.wikipedia.org/wiki/Atlas_(topology)#Transition_... although the quoted sentence does not make sense to me; transition maps connect different patches of one manifold. it's possible the "LLM space" gp is talking about is a parameter space of some nature each of whose points is a manifold, but that seems like a stretch

moralestapia 3y ago
You're right, that's a bit ambiguous. Instead of "all different manifolds" I should have written "all different ways to define a manifold". Also, now that I think about it, it may not necessarily be all of them. The important thing here is that, however you define them and their transition maps, you'll find they're very much alike. As if there was some sort of general structure that is highly preferred over others ...
- evertedsphere 3y ago
  "if you look at the (chart) transition maps for different atlases on this manifold, they tend to look similar for different atlases" what is the manifold here and what evidence do you have for this / what does it look like when you "define" them?
  - moralestapia 3y ago
    See, https://transformer-circuits.pub/2023/may-update/index.html#feature-manifolds https://transformer-circuits.pub/2023/may-update/index.html#.... They also reference, https://distill.pub/2020/circuits/equivariance/ https://distill.pub/2020/circuits/equivariance/. I'd also add, https://distill.pub/2021/multimodal-neurons/ https://distill.pub/2021/multimodal-neurons/.
codethief 3y ago
Yeah, that's the definition I was thinking of, too, but the sentence didn't make sense to me, either.