3 ms·

Completely different model sizes? Stable Diffusion is a ~1B parameter model. The 7B parameter LLAMA model would more comparable in RAM usage. It's not accurate

Completely different model sizes? Stable Diffusion is a ~1B parameter model. The 7B parameter LLAMA model would more comparable in RAM usage. It's not accurate to say "LLAMA needs 130 GB" because there's more than one LLAMA. The hard part in general is it often doesn't work very well to train small model sizes directly. You can train a very large one and distill it down, in some cases to only 1% of the original size while retaining 99%+ quality. So clearly the 1% size model exists. However training it directly usually doesn't work nearly as well. Best I can tell no one knows for sure why. Big tech cos are training the largest models as no one else has the hardware/power/money/etc to do so. SO they tend to release the most massive ones that only they can make. There's also a sect in the ML community that thinks "scale" is the answer to the universe... Then, either the tech cos or the community will make multiple other sizes of models and those are the ones that normies can use.

Helmut10001 4y ago
> There's also a sect in the ML community that thinks "scale" is the answer to the universe... Offtopic: Reminds me of Laurent Nottale and his controversial "Scale Relativity and Fractal Space-Time: A New Approach to Unifying Relativity and Quantum Mechanics"
espadrine 4y ago
Model sizes are adjusted to reach a given quality. Many papers justified empirically the use of more neurons allocated to language than to visual understanding, like Imagen[0] (figure 4.a). as it yields better results. Debatably, it is also true in the human brain. To me, it could highlight that there is more entropy in language than in images. After all, games have a similar property: some games, like noughts and crosses, or checkers, can have the same model architecture reach optimal play with smaller sizes than are necessary for chess and go[1]. It is certainly true that language has a lot of rules (grammatical, vocabular, syntactic) which are necessary to master it, but irrelevant to having a mental model of a scene; it just sequences a state of things into a stream of symbols. Maybe an additional entropy to learn in language which is absent in images includes motion and emotion. [0]: https://arxiv.org/pdf/2205.11487.pdf https://arxiv.org/pdf/2205.11487.pdf [1]: https://arxiv.org/pdf/2104.03113.pdf https://arxiv.org/pdf/2104.03113.pdf
dragonwriter 4y ago
> There's also a sect in the ML community that thinks "scale" is the answer to the universe The Basilisk Cult is a clear and influential example of this, but I thnk it extends beyond them.

Helmut10001 4y ago

> There's also a sect in the ML community that thinks "scale" is the answer to the universe... Offtopic: Reminds me of Laurent Nottale and his controversial "Scale Relativity and Fractal Space-Time: A New Approach to Unifying Relativity and Quantum Mechanics"

espadrine 4y ago

Model sizes are adjusted to reach a given quality. Many papers justified empirically the use of more neurons allocated to language than to visual understanding, like Imagen[0] (figure 4.a). as it yields better results. Debatably, it is also true in the human brain. To me, it could highlight that there is more entropy in language than in images. After all, games have a similar property: some games, like noughts and crosses, or checkers, can have the same model architecture reach optimal play with smaller sizes than are necessary for chess and go[1]. It is certainly true that language has a lot of rules (grammatical, vocabular, syntactic) which are necessary to master it, but irrelevant to having a mental model of a scene; it just sequences a state of things into a stream of symbols. Maybe an additional entropy to learn in language which is absent in images includes motion and emotion. [0]: https://arxiv.org/pdf/2205.11487.pdf https://arxiv.org/pdf/2205.11487.pdf [1]: https://arxiv.org/pdf/2104.03113.pdf https://arxiv.org/pdf/2104.03113.pdf

dragonwriter 4y ago

> There's also a sect in the ML community that thinks "scale" is the answer to the universe The Basilisk Cult is a clear and influential example of this, but I thnk it extends beyond them.