3 ms·

Haven't heard about the universal subspace hypothesis yet, so I appreciate the digression.

• thoughtpeddler 4mo ago
  Ya, super interesting research area the authors explored of basically trying to answer the question: "Is there a canonical/intrinsic way that concepts/representations/information are 'stored' in the universe/reality?". They tested that by performing "spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models ... by applying spectral decomposition techniques to the weight matrices of various architectures", and concluding that "deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces", showing that "neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain". Not just philosophically interesting but also has practical implications for being smarter about how to reuse models, model merging, developing more sustainable training and inference algos, etc. Paper source: https://arxiv.org/abs/2512.05117 https://arxiv.org/abs/2512.05117
  • Kye 4mo ago
    Maybe related: https://news.ycombinator.com/item?id=47322887 https://news.ycombinator.com/item?id=47322887
    • vessenes 4mo ago
      Yeah that layer looping is super interesting. Also, it’s memory friendly with the right inference harness.
  • gom_jabbar 4mo ago
    [dead]
  • vessenes 4mo ago
    Very Chomsky friendly. Interesting paper, thank you!