5 ms·

It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.

• speed_spread 2mo ago
  If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.
  • wolttam 2mo ago
    I think we will eventually reach a point where this is the case, but at the moment it seems like you can throw virtually any non-trivial use-case at a model today and end up being more satisfied with the results that a model tomorrow gives. I may just be closed minded as to what use-cases we have that current models are truly "good enough" (i.e. won't be dissatisfied when comparing results of today's model to tomorrow's model)
• nine_k 2mo ago
  Not so if it's embedded in something smart enough for its intended purpose. Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.
• deleted 2mo ago
  [deleted]
• anigbrowl 2mo ago
  This is only true for people who are solely focused on performance. There is absolutely a market for acceptable performance combined with predictability.
  • teraflop 2mo ago
    True, but predictability cuts both ways. We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time. Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash. What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware. Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it. [1]: https://openai.com/index/where-the-goblins-came-from/ https://openai.com/index/where-the-goblins-came-from/
    • noisy_boy 2mo ago
      > Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it. Does it though? Isn't that what CPUs are, very fast-not-so-clever computing brain surrounded by layers that protect it?
    • anigbrowl 2mo ago
      I see your argument but your example seems highly contrived. I can't think why you'd want to use something like this for something as dynamic as takeout ordering, where you might have to deal with bad customers, supply chain breakages, public health recalls, or any of many other probabilistic events. I think it's far more likely to see them used in safety critical applications where you need a capable model that can run on low power and doesn't have multiple layers of operating abstractions between the model and the hardware.
      • 2mo ago
• twobitshifter 2mo ago
  OTOH, people get a new iPhone every year and they are ok with it.
  • nomel 2mo ago
    How is that in any way related to a consumer device? This method doesn't reduce physical memory requirements, so still results in huge die area. This isn't a for-end-user thing, probably for decades.
    • twobitshifter 2mo ago
      Ok, how long until nvidia gives us a new GPU?
      • nomel 2mo ago
        I don't follow. How is that related? GPUs don't have fixed memory. You don't throw them away when you want to load a new model. NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.
        • twobitshifter 2mo ago
          Point being that hardware generations can be very quick and as updates get made GPUs go out of date and can’t run the latest models. All types of hardware consumer of otherwise are always improving, which means that tying a model the hardware is not going to lead to increased obsolescence, any more than the hardware itself does. If an AI is general purpose then there is no problem with only having that one model baked into the chip, when the benefit is a huge speed and efficiency improvement over running it on a GPU.
          • nomel 2mo ago
            I think there's some confusion about what's going on here. A GPU is general purpose, for inference sake. You can run any model that can fit in it. It will be obsolete, as all hardware eventually is, but a 3090 today is more useful than a 3090 two years ago, because small models have improved significantly. Hardware as a model can run exactly one model, ever. You can't try a fine tune, and can't try the new similarly sized model that's better than all then others you've ever tried. You can run exactly one set of weights, with the architecture it shipped with, because everything is fixed.
• askl 2mo ago
  Might be worse because it takes time between designing the silicon and having the first usable chips. So they're outdated the moment they hit the market or even before that.