4 ms·
Massive overreaction in my opinion (not financial advice). People are forgetting how much compute is being used for inference. This is going to be further acce
by Palmik 2y ago
Massive overreaction in my opinion (not financial advice).
People are forgetting how much compute is being used for inference. This is going to be further accelerated:
- Reasoning models are going to generate orders of magnitude more tokens.
- Synthetic an augmented data is more and more prevalent, and if you need to process pre-training scale dataset, you will need a lot of inference.
- etc.
- dumbfounder 2y agoNo one will ever need more than 640k tokens.
- physicsguy 2y agoDistillations are showing (and have been for a while) though that you don't necessarily need big hardware for decent inference. If that's the case, and you can run it on your Macbook trivially or even your smartphone.
- fspeech 2y agoInference can be done on much more diverse hardware.
- mrbungie 2y agoWhen put into context one could say the OG overreaction was thinking 500B USD in infrastructure investments and restarting nuclear plants were rational decisions rather than working on efficiency.