4 ms·
Every reply here forgets/overlooks the main reason for why this is not going to happen: The astronomical AI data center investments currently underway. Those pl
by teiferer 5mo ago
Every reply here forgets/overlooks the main reason for why this is not going to happen: The astronomical AI data center investments currently underway. Those place are not just for training. They are for inference too and the way all those investments are expected to eventually pay off. The whole AI sector of our industry depends on running models in these places.
- zozbot234 5mo agoThese astronomical AI data centers will be used for high-value inference with smarter models that really are too large for running locally. The investments will be fine once they pivot to that use. Currently available open models are not in that range.
- teiferer 5mo agoI don't buy that that will be a useful distinction. First of all, no AI model will say "I'm too smart for this question, I suggest you use a cheaper one so I don't make unnecessary money for my owner" or "I'm too dumb, so instead of hallucinating I'll suggest you go to the cloud and ask my smarter sibling". Second, there is no incentive in the market for tooling to evolve that way. There will be the illusion that some models will do that, similar to today (or maybe some harnesses rather) but nobody will willinglylet money sit on the table. These data centers are not being built to solve world hunger. They are built to ultimately hook you on more realistic fake bs youtube videos so you feel good while getting even more ads injected into your life.
- jononor 5mo agoDynamic routing is the usual name for the piece that orchestrates which LLM will be used, based on query complexity. There in an open source implementation as part of the vLLM project (and probably others), it is a field of active research in several universities and labs. It is also suspected that the frontier LLM providers might already be doing something like it behind the covers.