5 ms·
Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out
by yumraj 2mo ago
Given the fast churn of the models, how does it work out?
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
- sigmoid10 2mo agoI find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.
- deadbabe 2mo agoCan you give some examples of these tasks that require an hour or more of reasoning?
- xyzsparetimexyz 2mo agoThe recent maths prompts did. The 'you should find a breakthrough' one was several blocks of reasoning, each taking 90 minutes or so
- sigmoid10 2mo agoI've let it research and implement pretty complex stuff for my 3d engine. Like a realtime volumetric cloud rendering solution. The kind of stuff you can do yourself, but it would probably take weeks, if not months. I spent quite a bit discussing the design with it, knowing some details of how it should work from game dev conference talks. Once we agreed on a design and implementation plan with key steps and intermediaries, i just said "now implement it" and it happily designed algorithms, wrote and ran tests, generated and looked at screenshots and basically did everything you would expect from a really senior engine programmer for more than an hour of wall time. The end result was something that worked amazingly well after testing it myself and suggesting minor corrections that were my fault for not specifying them in the first place.
- prinny_ 2mo agoThey expect a sort of breakpoint at which each subsequent model version will only be marginally better than the previous ones, thus allowing them to retain their value for some time. Their business doesn’t work if each year the new model demolishes the previous one in terms of performance.
- deadbabe 2mo agoYou could take your silicon chip and have it re-etched only with model diffs for an upgraded version.
- XCSme 2mo agoI think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
- m463 2mo agosubscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.
- XCSme 2mo agoSo local personalized ads? Not sure if that's better or worse than online personalizaed ads...
- throwaway173738 2mo agoYou don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.
- XCSme 2mo agoIt was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)
- tyre 2mo agoThink of a HomePod. 99% (and likely much more) of what people are asking is super simple.
- yumraj 2mo agoMakes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them
- laweijfmvo 2mo agopretty much everything is “1 or more versions behind” by the time it comes out. the question is whether or not it’s still useful? at some point, presumably not every application will need the latest cutting edge huge model.
- brokencode 2mo agoAlready models have gotten really good at a lot of things. A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper. And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.
- christina97 2mo agoThere’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).
- craftkiller 2mo agoI think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.
- Maxion 2mo agoOr as subagents reading code or other text and summarizing them.
- nullbio 2mo agoPerfect for consumers. You buy it and then you need to buy a new one in a couple of years. If they can make them affordable they'll sell like hotcakes.
- etoxin 2mo agoAnd the second hand market. I'd love to see this integrated into motherboards like RAM. Someone could have a motherboard with 4 sticks of different AI with various models. Swap, change and trade.
- chorizo 2mo agoThat’s not going to be true forever. As models mature, we will hit diminishing returns. Major improvements will come annually rather monthly - matching the roughly annual release of new processors. Model ROM’s will likely get integrated into die packages just like DRAM now.
- pennomi 2mo agoI’m hoping for SNES style cartridges
- mrheosuper 2mo agoI'm still using Opus for most daily task because Fable is too expensive. If they begin etching Fable into silicon now and release it 2-3 years later, i can see the market for it
- noosphr 2mo agoThis is a feature for most local use cases. You don't want all your work flows to start failing because of a model update.
- casey2 2mo agoThere isn't a fast churn in the underlying pretrained model, nor RL. It's mostly orchestration around the model. Said another way you could just pretrain and RL for longer. Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.
- hahahaa 2mo agoThey still make 6502s right.
- cdolan 2mo agoWe have some processes running on models released a year ago (which we're updating, but still) The speed is incredible. It doesnt matter if you are ~30-300 days behind
- trklausss 2mo agoThe thing is, right now it is exploding because we are at the beginning of it. At some point, it will plateau at a specific level, and not that much quality will be gained. There is however leaps to make for efficiency. The same can be said about the CISC computer: yes, new processors introduce new instructions that do something slightly faster, you could still crunch that with an older processor. The real benefit comes in clock cycles (that's why Arm with a reduced set can compete with x86). Also: there are myriads of models, for myriads of tasks. Not all have the same development gains as we see for general purpose AI. If you etch those, you reduce your bill by factors down. It also democratises models: Instead of running them on a cloud server by some company, you can run them at home, for coding tasks, without the need of internet connection, etc.
- woodrowbarlow 2mo agoalso, as this scales, what would this mean for closed-weight hosted models? i imagine it's possible (but difficult) to re-derive model weights by de-lidding and inspecting the die... so will this only ever be used for open-weights models?