3 ms·
What fascinates me most about these LLMs and other machine-learning technology is how it is driven by the availability of GPUs. Essentially, everything has to b
by choeger 3y ago
What fascinates me most about these LLMs and other machine-learning technology is how it is driven by the availability of GPUs. Essentially, everything has to be carefully encoded into - differentiable - linear algebra operations for practical reasons.
On the other hand, there are modeling languages (Modelica being a prime example) in search of a principled target calculus. So for me the obvious question is: Should one strive to develop a modeling language that compiles to GPU- and learning-compatible operations? Would that help to improve the models as well as understand them?
- jacquesm 3y agoThat seems to be the wrong way around. We're constrained by the availability of GPUs because those are the processors of choice for this kind of solution. If we had a better one we'd be using it!
- thanatropism 3y agoWhat GP seems to be saying is that deep learning flourished because of the availability of GPUs, if fast SAT solving chips had appeared before.
- awegio 3y agoRelated: The Hardware Lottery (https://arxiv.org/abs/2009.06489 https://arxiv.org/abs/2009.06489) It goes both ways: processors are built to accelerate existing applications, and applications are built to run with the constraints of the hardware. This holds back progress towards better systems where both new kind of hardware and software is required. This is part of the reason why research into neural networks was so slow for a long time, and took off only after GPUs for 3d graphics came around.