4 ms·
Do you know of a good reference / primer for LLMs from a technical architecture perspective? I've been somewhat avoiding them, but after seeing MonadGPT -- I'm
by potsandpans 3y ago
Do you know of a good reference / primer for LLMs from a technical architecture perspective? I've been somewhat avoiding them, but after seeing MonadGPT -- I'm just too damn curious.
Ideally, I'd like to be able to have a "survey level" understanding of what goes into scaling these models, and what they're capable of at different levels of scale. For example, in the "introducing llama" page, they say
> Smaller, more performant models such as LLaMA enable others in the research community who don’t have access to large amounts of infrastructure to study these models, further democratizing access in this important, fast-changing field.
I'd like to be able to somewhat intelligently be able to discuss the tradeoffs here. What exactly does "smaller, more performant" mean in this context and how can we quantify the differences between models that demand larger infrastructure.