2 ms·
> It’s because weights are where the magic lies, for the most part. What's that supposed to mean? Weights are not independent of the underlying architecture.
by kleiba 3y ago
> It’s because weights are where the magic lies, for the most part.
What's that supposed to mean? Weights are not independent of the underlying architecture.
- dartos 3y agoThey’re not, but the difference between the architecture outputting random garbage and impressive text is all in the weights.
- Nevermark 3y agoAnd a fitting architecture can save orders of magnitude in numbers of weight and training time. There isn’t one that’s more important than the other. There are no weights at all without an architecture. And any architecture improvements that pay off have a multiplier effect on efficiency. Nobody on Earth had the resources to get the performance they are getting on these big models with just a default fully connected layer architecture.