3 ms·

Solving a set of analogous problems and then refining from those doesn't sound all that different from the general idea of adding another layer.

• kromem 3y ago
  At a certain point, I hope NLP research finally gets that the neurological processes we're attempting to model aren't a single giant unified network either and starts looking at layering models as a way to achieve success and not simply layers within a single model. Jailbreaking/hallucination issues? Maybe there needs to be the equivalent of additional prefrontal cortex impulse control. We know from split brain studies that the apparent unity of personality/ego/consciousness is anything but unified. I have a suspicion we're going to end up getting much more traction in the future with multiple smaller models layered together (more than even GPT-4's rumored MOE) than with a single giant model trying to do everything all at once. "Moar layers" as the trick to achieve performance may need to be broadened from adding layers of neurons in a single network to adding layers of networks in a mesh, and potentially even layers of meshes.