4 ms·

I tried Kimi K3 on a task I've done with every other model I use regularly (https://swelljoe.com/post/i-let-every-agent-implement-its-own-flar-backend/ https://

I tried Kimi K3 on a task I've done with every other model I use regularly (https://swelljoe.com/post/i-let-every-agent-implement-its-own-flar-backend/ https://swelljoe.com/post/i-let-every-agent-implement-its-ow...) and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan. I only have the $20 plan from OpenAI and the same task, with a lot of the same implementation details as Kimi Code, only took a few minutes and consumed almost none of the 5 hour limit. Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare, but when I sat down to add Kimi Code to flar, it was because I wanted to try it on some real work and then couldn't do any, because usage was nearly gone after the trivial task...no other ~$20 subscription I have has felt that tight before. So, it was really slow to complete the task and seemingly much more expensive than every other model I'd tried. Maybe bad luck. Maybe it'll do better on other tasks. I wouldn't know as I was out of usage when I had time to try. It did find a bug that Gemini 3.5 Flash introduced unprompted, though, so it has that going for it.

fmbb 3mo ago
Is Kimi K3 subsidized as hard as the other models out there?
- brookst 3mo ago
  Does it matter? As an end user I really only care about 1) how much I can do in a week, and 2) how long each task takes. Subsidies would affect 1, but not 2. But if some VC wants to subsidize my Claude or Codex or whatever, awesome.
  - recursive 3mo ago
    It doesn't matter if you can switch easily. It might matter if there are barriers to switching.
  - mips_avatar 3mo ago
    The more important question than subsidy is what is the tokenomics of running the model. If it's inefficient to run on an nvl72 cluster (or whatever the heck has enough vram to run a 3T parameter model), and k3 isn't very token efficient, then it might not be that compelling of an open weights model.
    - foobar10000 3mo ago
      3T at nxfp4 (which is most of it) is only 1.5TB of vram - so 8x288GB B300 or MI355 will do it if you are careful with context - maybe dp-attn? Certainly not TP. 2 of those together can easily serve it. The new AMD MI400 are at 400GB+ each, so 8x of them will nicely fit with KV to spare.
  - popalchemist 3mo ago
    Subsidization could affect both of those. If you have $200B in the bank you can afford to throw massive compute at every single request; if you are less well funded, you might throttle more aggressively. Additionally that same VC could be (read: is always) spent on developing the harness, and other infrastructure around the model, not just the model itself. So it's apples-to-oranges when comparing a relatively new model to established competitors (i.e. OpenAI @ $900B funding vs Moonshot/Kimi's $30B FYI) because every new model they release is judged on "performance" which is not strictly speaking derived solely from the model. It's possible Moonshot could get similar performance over time as the build out the rest of the infrastructure. We have no way of knowing how much of OpenAI/Anthropic's success is due to the model vs intelligent tooling built on top of it.
- SwellJoe 3mo ago
  Not sure how the economics work for the Chinese models, but DeepSeek did the same task for a dime.
  - VulgarExigency 3mo ago
    In my opinion, for the vast majority of use cases, DeepSeek is still the most cost-effective model by a mile. $10 feels like it lasts forever.
    - SwellJoe 3mo ago
      Yep, with Reasonix, DeepSeek is free real estate. Seems to just go and go for pennies. And, DeepSeek is what I use for any task that works best with an API. It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry, and it's good enough to where I rarely have to follow up with a more expensive model or manually fix things. It's been alleged they're releasing an update to DeepSeek V4 Pro soon that improves it, which likely makes it a good fit for even more kinds of problems. It remains my favorite of the Chinese models, it's so cheap and cheerful. And, is also less aggressively censored than some of them.
      - ignoramous 3mo ago
        > It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry I use DeepSeek v4 Flash & MiMo v2.5 Pro. Prefer the latter over DeepSeek v4 Pro because it costs the same while being equally good & less chattier for coding workloads. Although, I've begun experimenting with Hy3 (as an in-between Flash & Pro) & GLM 5.2 (for long-horizon tasks).
    - Gigachad 3mo ago
      Maybe I’m not pushing them hard enough but I use Claude opus at work and deepseek v4 flash at home and they both seem about as capable. While deepseek is borderline free.
- j2j8 3mo ago
  From the end of the month it will be served profitably by other providers around the world, like Kimi K2.6
  - LUmBULtERA 3mo ago
    Sure, but at those API rates it doesn't beat the subscription plans for OpenAI.
mips_avatar 3mo ago
It would be really interesting to redo the public benchmarks for kimi k3 but token normalize the costs. Ok so maybe k3 beats fable on terminal bench, but how many tokens did it use?
techjamie 3mo ago
In my experience, Kimi just tends to think a lot, with the main thing that takes up a lot of space is it constantly second-guessing itself. I've watched it do paragraph after paragraph of "Wait, actually..." while it stumbled and used a ton of tokens on one small detail of what it was asked to do. Though I also gave GLM 5.2 a task to port some JS code to Python to test it, and in my experience it doesn't second guess as bad as Kimi does, but it really did there. It kept doing web searches and second guessing tons of tiny little things, using $0.25 of API spend in total to port about ~50 lines of JavaScript. It did produce an error the first run, but on second run it gave me a program that ran. I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences. I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.
- mikojan 3mo ago
  Interesting. Could you not tell it to not second-guess itself (on minor issues)?
  - manmal 3mo ago
    Fighting against the weights is often a lost cause. For any model.
ignoramous 3mo ago
> tried Kimi K3 on a task I've done with every other model I use regularly and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty. > Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use. In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai. [0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.
esafak 3mo ago
OpenAI measures token efficiency. Look at the API cost charts in their announcement: https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/
dehrmann 3mo ago
> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.
- dgellow 3mo ago
  You call it goofy, in a different context we would call that a dark pattern, shady, prone to fraud
- ekidd 3mo ago
  AI subscription pricing was fine when it was $100/month for some opaque 5 hour token budget I don't think I ever used, not even that one day where I coded for 14 hours non-stop using Fable. But like most people with low token usage, I had a human in the loop and and I didn't use workflows with swarms of agents. Now, of course, the plan is to remove Fable from the subscription. To paraphrase Darth Vader, they have altered the deal. Pray they do not alter it further.
  - steve-atx-7600 3mo ago
    Gpt 5.6 is still like this at least for the $200/month option. It’s also always faster than fabel. Fabel might be able to do some things better but I don’t have time to constantly wait and find out.
  - arcanemachiner 3mo ago
    They're probably not going to remove Fable. They just extended it for another month.
Aurornis 3mo ago
With the obligatory disclaimer that I’m impressed with what open weight models can do, I have the same experience with all of them. The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems. There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe. There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest. If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.
jug 3mo ago
Yeah I've noted this behavior with best in class open weight models. They said K3 would have token efficiency improvements and I was hoping especially solving the thinking loop issue that plagued K2.x but even if this release helped somewhat, it looks like we still have a long way to go here... I'm not sure what's up here but I suppose lacking finetuning quality. What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla. Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories. What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost. I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.
- SwellJoe 3mo ago
  Yeah, I'm finding I end up switching to Codex and GPT 5.6 a lot lately because I've either run out of Fable usage or Fable refused to do the task. Most recently it refused to work on a WiFi configuration UI for a robot. No idea why it thought that was related to security, biology, or some other sensitive topic. They've hobbled it with guardrails that are overzealous and now there's a big opening in the market. Fable may be the best, but if it won't do the job half the time, it stops being my go to model as I don't want to waste time only to find it refuses halfway through.
osti 3mo ago
Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.
- segmondy 3mo ago
  how do we get chinese phone numbers?
  - xyzsparetimexyz 3mo ago
    esim
    - osti 3mo ago
      Nah that won't work. I don't know tbh, I just used someone else's number.
      - segmondy 3mo ago
        how do I find someone to use their number when I'm not in china and don't know anyone in china?
    - tommydee 3mo ago
      not gonna happen. China has strict real name verification for SIMs.
- xyzsparetimexyz 3mo ago
  Wow! Does it accept a foreign alipay/wechat pay account?
  - nonethewiser 3mo ago
    Those exist?
ffsm8 3mo ago
Earlier today I made Claude code implement a feature with fable. It worked roughly 60 minutes and used around 30% of my 100€ subs 5h sessions. Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that. Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so
- fibonacci112358 3mo ago
  /code-review in Claude Code spawns a lot of sub-agents (counted like 8 once), each looking at the code from some certain aspect (like correctness, maintainability, duplication, testing, etc). It eats tokens like crazy doing that, but also covers quite a lot. The default code review in Codex does far less (feels like it's only correctness) and doesn't uses subagents. Actually I made a skill for Codex that does a review closer to what Claude does by default, but using like 4-5 agents and some being cheaper models/less than xhigh reasoning. I'm getting pretty nice reviews with that that cover more than just correctness.
  - CharlesW 3mo ago
    You can do `/code-review low` to more closely match Codex's default behavior.
- manmal 3mo ago
  Read the session file of the reviewer tomorrow. Sonnet should suffice.
j-bos 3mo ago
Aren't they still locking reasoning to "max" pending adjustments to support shorter reasoning levels.
- nonethewiser 3mo ago
  Why would they do that? Sounds terrible
- SwellJoe 3mo ago
  It let's me choose different thinking levels in Kimi Code. Not sure if it actually works, yet, but it says "Thinking set to high." when I change it from max.
- j-bos 3mo ago
  Found it! > At launch, Kimi K3 will use max thinking effort by default, with low- and high-effort modes to be introduced in subsequent updates https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3
jrflo 3mo ago
We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.
- resonious 3mo ago
  The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it. But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.
  - a_wild_dandan 3mo ago
    Wow, you weren't kidding. I looked at their chart, and the cost-per-task for Fable is more than double Sol's. And DeepSeek absolutely stomps. Four cents per-task vs Sol's $1 and Fable's $3. I might need to check out DeepSeek more. I had no idea the difference was this obscene. Makes me wonder if something's off with the benchmark. A 70x cost reduction vs. Fable seems too good to be true.
    - SwellJoe 3mo ago
      In my benchmarks of security auditing abilities of models, DeepSeek was roughly an order of magnitude cheaper than either GPT 5.5 or Opus 4.8, less than ten cents per task vs. roughly a buck each for the best American models at the time. GPT 5.5 Pro was ~230x at almost $23 per task. DeepSeek is my go-to when I need an API, and local Gemma 4 won't do because it's either too slow or not capable enough. DeepSeek isn't at the frontier but it's good enough for a lot of things, very cheap, and quite fast. Flash is even faster and cheaper, and still better than anything I can host locally.
  - MrBuddyCasino 3mo ago
villish 3mo ago
Do models know when they're being benchmarked? I also used K3 briefly and noticed it spent a very long time thinking and obviously a lot of tokens. However I've seen some benchmarks say it uses fewer than fable which hasn't been my experience.
lebovic 3mo ago
Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1]. When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree. I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings. [1]: https://platform.kimi.ai/docs/guide/use-thinking-effort https://platform.kimi.ai/docs/guide/use-thinking-effort

fmbb 3mo ago

Is Kimi K3 subsidized as hard as the other models out there?

brookst 3mo ago
Does it matter? As an end user I really only care about 1) how much I can do in a week, and 2) how long each task takes. Subsidies would affect 1, but not 2. But if some VC wants to subsidize my Claude or Codex or whatever, awesome.
- recursive 3mo ago
  It doesn't matter if you can switch easily. It might matter if there are barriers to switching.
- mips_avatar 3mo ago
  The more important question than subsidy is what is the tokenomics of running the model. If it's inefficient to run on an nvl72 cluster (or whatever the heck has enough vram to run a 3T parameter model), and k3 isn't very token efficient, then it might not be that compelling of an open weights model.
  - foobar10000 3mo ago
    3T at nxfp4 (which is most of it) is only 1.5TB of vram - so 8x288GB B300 or MI355 will do it if you are careful with context - maybe dp-attn? Certainly not TP. 2 of those together can easily serve it. The new AMD MI400 are at 400GB+ each, so 8x of them will nicely fit with KV to spare.
- popalchemist 3mo ago
  Subsidization could affect both of those. If you have $200B in the bank you can afford to throw massive compute at every single request; if you are less well funded, you might throttle more aggressively. Additionally that same VC could be (read: is always) spent on developing the harness, and other infrastructure around the model, not just the model itself. So it's apples-to-oranges when comparing a relatively new model to established competitors (i.e. OpenAI @ $900B funding vs Moonshot/Kimi's $30B FYI) because every new model they release is judged on "performance" which is not strictly speaking derived solely from the model. It's possible Moonshot could get similar performance over time as the build out the rest of the infrastructure. We have no way of knowing how much of OpenAI/Anthropic's success is due to the model vs intelligent tooling built on top of it.
SwellJoe 3mo ago
Not sure how the economics work for the Chinese models, but DeepSeek did the same task for a dime.
- VulgarExigency 3mo ago
  In my opinion, for the vast majority of use cases, DeepSeek is still the most cost-effective model by a mile. $10 feels like it lasts forever.
  - SwellJoe 3mo ago
    Yep, with Reasonix, DeepSeek is free real estate. Seems to just go and go for pennies. And, DeepSeek is what I use for any task that works best with an API. It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry, and it's good enough to where I rarely have to follow up with a more expensive model or manually fix things. It's been alleged they're releasing an update to DeepSeek V4 Pro soon that improves it, which likely makes it a good fit for even more kinds of problems. It remains my favorite of the Chinese models, it's so cheap and cheerful. And, is also less aggressively censored than some of them.
    - ignoramous 3mo ago
      > It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry I use DeepSeek v4 Flash & MiMo v2.5 Pro. Prefer the latter over DeepSeek v4 Pro because it costs the same while being equally good & less chattier for coding workloads. Although, I've begun experimenting with Hy3 (as an in-between Flash & Pro) & GLM 5.2 (for long-horizon tasks).
  - Gigachad 3mo ago
    Maybe I’m not pushing them hard enough but I use Claude opus at work and deepseek v4 flash at home and they both seem about as capable. While deepseek is borderline free.
j2j8 3mo ago
From the end of the month it will be served profitably by other providers around the world, like Kimi K2.6
- LUmBULtERA 3mo ago
  Sure, but at those API rates it doesn't beat the subscription plans for OpenAI.

mips_avatar 3mo ago

It would be really interesting to redo the public benchmarks for kimi k3 but token normalize the costs. Ok so maybe k3 beats fable on terminal bench, but how many tokens did it use?

techjamie 3mo ago

In my experience, Kimi just tends to think a lot, with the main thing that takes up a lot of space is it constantly second-guessing itself. I've watched it do paragraph after paragraph of "Wait, actually..." while it stumbled and used a ton of tokens on one small detail of what it was asked to do. Though I also gave GLM 5.2 a task to port some JS code to Python to test it, and in my experience it doesn't second guess as bad as Kimi does, but it really did there. It kept doing web searches and second guessing tons of tiny little things, using $0.25 of API spend in total to port about ~50 lines of JavaScript. It did produce an error the first run, but on second run it gave me a program that ran. I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences. I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.

mikojan 3mo ago
Interesting. Could you not tell it to not second-guess itself (on minor issues)?
- manmal 3mo ago
  Fighting against the weights is often a lost cause. For any model.

ignoramous 3mo ago

> tried Kimi K3 on a task I've done with every other model I use regularly and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty. > Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use. In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai. [0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.

esafak 3mo ago

OpenAI measures token efficiency. Look at the API cost charts in their announcement: https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/

dehrmann 3mo ago

> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.

dgellow 3mo ago
You call it goofy, in a different context we would call that a dark pattern, shady, prone to fraud
ekidd 3mo ago
AI subscription pricing was fine when it was $100/month for some opaque 5 hour token budget I don't think I ever used, not even that one day where I coded for 14 hours non-stop using Fable. But like most people with low token usage, I had a human in the loop and and I didn't use workflows with swarms of agents. Now, of course, the plan is to remove Fable from the subscription. To paraphrase Darth Vader, they have altered the deal. Pray they do not alter it further.
- steve-atx-7600 3mo ago
  Gpt 5.6 is still like this at least for the $200/month option. It’s also always faster than fabel. Fabel might be able to do some things better but I don’t have time to constantly wait and find out.
- arcanemachiner 3mo ago
  They're probably not going to remove Fable. They just extended it for another month.

Aurornis 3mo ago

With the obligatory disclaimer that I’m impressed with what open weight models can do, I have the same experience with all of them. The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems. There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe. There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest. If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.

jug 3mo ago

Yeah I've noted this behavior with best in class open weight models. They said K3 would have token efficiency improvements and I was hoping especially solving the thinking loop issue that plagued K2.x but even if this release helped somewhat, it looks like we still have a long way to go here... I'm not sure what's up here but I suppose lacking finetuning quality. What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla. Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories. What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost. I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.

SwellJoe 3mo ago
Yeah, I'm finding I end up switching to Codex and GPT 5.6 a lot lately because I've either run out of Fable usage or Fable refused to do the task. Most recently it refused to work on a WiFi configuration UI for a robot. No idea why it thought that was related to security, biology, or some other sensitive topic. They've hobbled it with guardrails that are overzealous and now there's a big opening in the market. Fable may be the best, but if it won't do the job half the time, it stops being my go to model as I don't want to waste time only to find it refuses halfway through.

osti 3mo ago

Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.

segmondy 3mo ago
how do we get chinese phone numbers?
- xyzsparetimexyz 3mo ago
  esim
  - osti 3mo ago
    Nah that won't work. I don't know tbh, I just used someone else's number.
    - segmondy 3mo ago
      how do I find someone to use their number when I'm not in china and don't know anyone in china?
  - tommydee 3mo ago
    not gonna happen. China has strict real name verification for SIMs.
xyzsparetimexyz 3mo ago
Wow! Does it accept a foreign alipay/wechat pay account?
- nonethewiser 3mo ago
  Those exist?

ffsm8 3mo ago

Earlier today I made Claude code implement a feature with fable. It worked roughly 60 minutes and used around 30% of my 100€ subs 5h sessions. Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that. Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so

fibonacci112358 3mo ago
/code-review in Claude Code spawns a lot of sub-agents (counted like 8 once), each looking at the code from some certain aspect (like correctness, maintainability, duplication, testing, etc). It eats tokens like crazy doing that, but also covers quite a lot. The default code review in Codex does far less (feels like it's only correctness) and doesn't uses subagents. Actually I made a skill for Codex that does a review closer to what Claude does by default, but using like 4-5 agents and some being cheaper models/less than xhigh reasoning. I'm getting pretty nice reviews with that that cover more than just correctness.
- CharlesW 3mo ago
  You can do `/code-review low` to more closely match Codex's default behavior.
manmal 3mo ago
Read the session file of the reviewer tomorrow. Sonnet should suffice.

j-bos 3mo ago

Aren't they still locking reasoning to "max" pending adjustments to support shorter reasoning levels.

nonethewiser 3mo ago
Why would they do that? Sounds terrible
SwellJoe 3mo ago
It let's me choose different thinking levels in Kimi Code. Not sure if it actually works, yet, but it says "Thinking set to high." when I change it from max.
j-bos 3mo ago
Found it! > At launch, Kimi K3 will use max thinking effort by default, with low- and high-effort modes to be introduced in subsequent updates https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3

jrflo 3mo ago

We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.

resonious 3mo ago
The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it. But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.
- a_wild_dandan 3mo ago
  Wow, you weren't kidding. I looked at their chart, and the cost-per-task for Fable is more than double Sol's. And DeepSeek absolutely stomps. Four cents per-task vs Sol's $1 and Fable's $3. I might need to check out DeepSeek more. I had no idea the difference was this obscene. Makes me wonder if something's off with the benchmark. A 70x cost reduction vs. Fable seems too good to be true.
  - SwellJoe 3mo ago
    In my benchmarks of security auditing abilities of models, DeepSeek was roughly an order of magnitude cheaper than either GPT 5.5 or Opus 4.8, less than ten cents per task vs. roughly a buck each for the best American models at the time. GPT 5.5 Pro was ~230x at almost $23 per task. DeepSeek is my go-to when I need an API, and local Gemma 4 won't do because it's either too slow or not capable enough. DeepSeek isn't at the frontier but it's good enough for a lot of things, very cheap, and quite fast. Flash is even faster and cheaper, and still better than anything I can host locally.
- MrBuddyCasino 3mo ago

villish 3mo ago

Do models know when they're being benchmarked? I also used K3 briefly and noticed it spent a very long time thinking and obviously a lot of tokens. However I've seen some benchmarks say it uses fewer than fable which hasn't been my experience.

lebovic 3mo ago

Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1]. When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree. I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings. [1]: https://platform.kimi.ai/docs/guide/use-thinking-effort https://platform.kimi.ai/docs/guide/use-thinking-effort