3 ms·

I'm not surprised, a space with 175B parameters will be very sparse.

• raphlinus 6y ago
  It's not obvious to me that this is the case. Chris Olah and others talk about "superposition" as a mechanism to explain "polysemantic" neurons that arise in image classifiers. To me, that suggests (using very vague, hand-wavy terms) that the optimization process is attempting to pack in as many concepts as possible into the finite parameter space. Certainly the scaling of GPT-3 suggests that these larger models are not necessarily any sparser than smaller ones. [1]: https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/
  • ozorOzora 6y ago
    That is fascinating, thanks for the link.