3 ms·

So all billons spent finding tricks and architectures that perfom well haven't resulted in any durable structures in contemporary LLMs? Each new training from

• Ukv 2y ago
  > Each new training from scratch is a perfect blank slate [...]? I don't think training runs are done entirely from scratch. Most training runs in practice will start from some pretrained weights or distill an existing model - taking some model pretrained on ImageNet or Common Crawl and fine-tuning it to a specific task. But even when the weights are randomly initialized, the hyperparameters and architectural choices (skip connections, attention, ...) will have been copied from previous models/papers by what performed well empirically, sometimes also based on trying to transfer our own intuition (like stacking convolutional layers as a rough approximation of our visual system), and possibly refined/mutated through some grid search/neural architecture search on data.