4 ms·
Something I find interesting is that in the last year or so, the talk around AI shifted from the model architecture to the trained model. People talk about Mist
by andy99 3y ago
Something I find interesting is that in the last year or so, the talk around AI shifted from the model architecture to the trained model. People talk about Mistral 7B e.g., not transformer with rotary position embedding and gelu feed forward network (I don't know Mistral's architecture).
Contrast this to a few years ago we'd talk about Resnet or Retina-net or whatever, not so much about the facebook pertain on Image-net when describing the model.
In 2018 I remember hearing "architecture is the new feature engineering" (mostly meaning over-fitting I undetstood). Now it's all (mostly) about the dataset and training, the architecture, in 2023, was a minor detail. I personally think architecture will make a comeback soon.
- dartos 3y agoIt’s because weights are where the magic lies, for the most part. Annnd nowadays tools like transformers can automatically select the architecture based on the name of the weights, so when interacting with LLMs, you generally just name the weights. Also MANY models are just using the llama or llama2 architecture.
- kleiba 3y ago> It’s because weights are where the magic lies, for the most part. What's that supposed to mean? Weights are not independent of the underlying architecture.
- dartos 3y agoThey’re not, but the difference between the architecture outputting random garbage and impressive text is all in the weights.
- Nevermark 3y agoAnd a fitting architecture can save orders of magnitude in numbers of weight and training time. There isn’t one that’s more important than the other. There are no weights at all without an architecture. And any architecture improvements that pay off have a multiplier effect on efficiency. Nobody on Earth had the resources to get the performance they are getting on these big models with just a default fully connected layer architecture.
- cma 3y agoTransformers self select the architecture based on the name of the weights? What does this mean?
- dartos 3y agoSorry, I meant huggingface transformers (the python package) The name is a little confusing, since the package supports non transformers models as well (like RWKV) I think HN took out the huggingface emoji from my post
- bugglebeetle 3y agoIIRC Mistral’s architecture is llama-2. They just trained it from scratch with undisclosed data and techniques.
- ilaksh 3y agoI think it did with Mixtral and MoE. I think people talk about the models because the architectural details are generally inaccessible for us since we don't have sufficient training (which it requires quite a lot to have an intuitive understanding I believe). Whereas the models are easily downloaded and tested.
- xiphias2 3y agoIt probably means that AI is getting more use outside technical circles. I would definitely include Mamba state space models and of course would prefer a technical review over a corporate review of the year.
- YetAnotherNick 3y agoFor all we know, Mamba could be overhyped and could only work for small models. Remember retnet[1] which said the same thing, equal to transformer in linear time. [1]: Retentive Network: A Successor to Transformer for Large Language Models: https://arxiv.org/abs/2307.08621 https://arxiv.org/abs/2307.08621
- YetAnotherNick 3y agoEven in Resnet era, data was the most relevant for accuracy. Google had this internal JFT data, training on which improves imagenet score much larger than different architecture. Also most of the gains after that came from data augmentation techniques or bigger model rather than new architecture.