3 ms·
It’s because weights are where the magic lies, for the most part. Annnd nowadays tools like transformers can automatically select the architecture based on th
by dartos 3y ago
It’s because weights are where the magic lies, for the most part.
Annnd nowadays tools like transformers can automatically select the architecture based on the name of the weights, so when interacting with LLMs, you generally just name the weights.
Also MANY models are just using the llama or llama2 architecture.
- kleiba 3y ago> It’s because weights are where the magic lies, for the most part. What's that supposed to mean? Weights are not independent of the underlying architecture.
- dartos 3y agoThey’re not, but the difference between the architecture outputting random garbage and impressive text is all in the weights.
- Nevermark 3y agoAnd a fitting architecture can save orders of magnitude in numbers of weight and training time. There isn’t one that’s more important than the other. There are no weights at all without an architecture. And any architecture improvements that pay off have a multiplier effect on efficiency. Nobody on Earth had the resources to get the performance they are getting on these big models with just a default fully connected layer architecture.
- cma 3y agoTransformers self select the architecture based on the name of the weights? What does this mean?
- dartos 3y agoSorry, I meant huggingface transformers (the python package) The name is a little confusing, since the package supports non transformers models as well (like RWKV) I think HN took out the huggingface emoji from my post