3 ms·
The base model is likely to have lots of genetic determined stuff that is there before any training: some animals like horses can walk and perceive/navigate ver
by cma 3y ago
The base model is likely to have lots of genetic determined stuff that is there before any training: some animals like horses can walk and perceive/navigate very soon after being born. There's also an innate fear of snake shapes in most mammals.
- exe34 3y agoAnother one that's bothered me is how our models so often start from scratch (aside from fine tuning or transfer learning) - I wish we could just start with say a smaller model with low resolution and grow it into a higher resolution one, a small LLM that can be grown into different specialisations (which we do with fine tuning - but do they all need the same number of layers, size of each layer, etc).