4 ms·

well that's exactly the point -- no such result is available for language models.

• verdverm 2y ago
  There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/awesome-llm-interpretability https://github.com/JShollaj/awesome-llm-interpretability
  • wwarner 2y ago
    Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong. I think this is a great paper.
    • verdverm 2y ago
      yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines (drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))
      • wwarner 2y ago
        I suppose, except that for a model of 7B parameters, the number of combinations of dropout that you'd be analyzing is 7B factorial. More importantly, dropout has loss minimization to guide it during training, whereas understanding how a model changes when you edit a few weights is a very broad question.
        • verdverm 2y ago
          the analysis is more akin to analyzing with & without dropout, where a common number is to drop a random 50% of connections during a pass for training, thus forcing the model to not rely on specific nodes or connections When you look at a specific input, you can look to see what gets activated or not. Orthogonal but related ideas for inspecting the activations to see effects