3 ms·

If I understand correctly, threat model here seems to be to protect against accidental issues that would impact performance, but doesn't cover malicious actor.

• j-bos 6mo ago
  Seems like a great challenge for all these systems, see fromtier labs serving quants when under hesvy load.
• gpm 6mo ago
  Yes and no. For a truly malicious actor, you're right. But it shifts it from "well we aren't obviously committing fraud by quantizing this model and not telling people" to "we're deliberately committing fraud by verifying our deployment with one model and then serving customer requests with another". I suspect there's a lot of semi-malicious actors who are only happy to do the former.
• nulltrace 6mo ago
  Catching accidental drift is still worth a lot. It's basically the same idea as performance regression tests in CI, nobody writes those because they expect sabotage. It's for the boring stuff, like "oops, we bumped a dep and throughput dropped 15%". If someone actually goes out of their way to bypass the check, that's a pretty different situation legally compared to just quietly shipping a cheaper quant anyway.
  • jychang 6mo ago
    Yeah, the threat model is nonexistent. Most people use a dozen or so well known providers, who have no incentives to so obviously cheat.
  • KeplerBoy 6mo ago
    Also it's not just about running an obviously worse quant. Running different GPU kernels / inference engines also matters. It's easy to write an implementation that is faster and thus cheaper but numerically much noisier / less accurate.
• frogperson 6mo ago
  Providers like OpenRouter default to the cheapest provider. They are often cheap because they are rediculously quantized and tuned for throughput, not quality. This is probably kimi trying to protect their brand from bargain basement providers that dont properly represent what the models are capable of.
  • latchkey 6mo ago
    > This is probably kimi trying to protect their brand from bargain basement providers that dont properly represent what the models are capable of. I'm curious what exactly they mean by this... "because we learned the hard way that open-sourcing a model is only half the battle."
    • HarHarVeryFunny 6mo ago
      I'd take it at face value. Since they release open weights they would appear to genuinely want other providers to serve this as well as themselves, but the benefit of this depends on it being served accurately.
      • latchkey 6mo ago
        I agree, but how about some details.
        • Onavo 6mo ago
          Kimi, GLM, and Minimax are the "Big Three" of open source Chinese AI startups. There's also Qwen and DeepSeek but they are all subsidized by other lines of business. The Chinese AI models are generally 5-6 months behind high end SOTA western models (and as of the time of this comment it's Opus 4.7 and ChatGPT 5.4 Thinking, it's rumored however that the Mythos and Spud codename models are even better). To gain market share, the Chinese startup use open source as a distribution strategy and essentially made mid-high end AI a commodity. The best models are still Western but for any application that doesn't require the highest performance in the market or if there's a need for extensive customization or alignment (imagine if you are an oil rich petro state and you don't want your national AI strategy to be tied to liberal international order ideology). It creates a lot of pricing pressure on the low and mid end, and it's also why Anthropic is desperately trying to go full B2B instead. However if the third parties hosting the Chinese models at near cost doesn't perform good quality control, it ruins the strategy because customers are not inclined to use chinese models anymore (and first party hosting on chinese infrastructure is out of the question because of geopolitical reasons, so everybody hides behind the polite fiction of using resellers like OpenRouter, Fal.ai, Wavespeed, fireworks AI etc.).
• cuki288 6mo ago
  [dead]