4 ms·

I guess it's more performant to stuff in a bigger system prompt now that models can support larger input sizes

• cubefox 2mo ago
  I would expect this only to be true for linear architectures like Mamba or Gated DeltaNet. Transformers and hybrid architectures do not have constant compute cost per token.
  • monkpit 2mo ago
    Performant could certainly mean “higher performing” and not “quicker”.