3 ms·

What's with the deceptive graph on top? Not including Astra can't have been an oversight, did the model compare poorly to it?

Iolaum 14d ago
I wonder if that means that SpaceX evals show that they consider astra better than fable or that they hate Sam&co so much they don't want to show their stuff.
- babelfish 14d ago
  this is exactly it.
- Jcampuzano2 14d ago
  https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/ https://openai.com/index/our-decision-on-cursor-following-it... Its because of this. You can't use Astra in Cursor, and cursorbench uses cursor as the harness. They can't actually benchmark it using their harness hence why its not included.
  - babelfish 14d ago
    They have Astra in other benchmarks lower on the page. They just don't want to show it winning
    - Jcampuzano2 14d ago
      The chart is cursorbench though and they asked about the "deceptive graph"
  - ryeguy 14d ago
    They can benchmark it because you can use an openai api key with cursor. Astra is just not included in the cursor plan.
- bluecalm 14d ago
  Elon posted on X that Grok 4.7 is behind Claude and OpenAI for agentic coding: https://x.com/elonmusk/status/2102082011233931762?s=20 https://x.com/elonmusk/status/2102082011233931762?s=20 so it's likely about usage in Cursor specifically.
Jcampuzano2 14d ago
https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/ https://openai.com/index/our-decision-on-cursor-following-it... This explains why. Mentioned in another comment, but cursorbench explicitly tests with Cursor as the harness, and OpenAI doesn't allow them to use Astra in Cursor.
- kristofferR 14d ago
  That's not accurate. OpenAI doesn't allow Grok to provide Astra to Cursor customers anymore, but it doesn't ban anyone from using Astra via alternative harnesses. If Cursor wanted to include Astra in CursorBench nothing would stop them, they could easily have spent half an hour vibecoding in OpenAI API key support - if it hadn't been convenient to neglect to do that.
  - andsoitis 14d ago
    Even if they could do that (workaround to include Astra in CursorBench), that has no practical consequences for Cursor users and that's what I as a Cursor user (what I use for dev, though I use ChatGPT for non-dev stuff) care about.
    - kristofferR 14d ago
      It would make the benchmark way better obviously, by showing how their new model compares to their competitors, the whole point of benchmarks and graphs.
      - Jcampuzano2 14d ago
        The point of Cursor Bench is to show how models perform in Cursor. If 99% of their users won't be able to access a model unless they go out of their way to include setup an API key for it (which would be insanely expensive with Astra), why would they include it in the benchmark?
- user43928 14d ago
scottyah 14d ago
Deceptive? An extremely quick google search would answer your question. OpenAI pulled out of Cursor before they released Astra so it never got that benchmark.
- kristofferR 14d ago
  Pulled out from letting them resell Astra access, that's not a limitation on running a benchmark.

Iolaum 14d ago

I wonder if that means that SpaceX evals show that they consider astra better than fable or that they hate Sam&co so much they don't want to show their stuff.

babelfish 14d ago
this is exactly it.
Jcampuzano2 14d ago
https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/ https://openai.com/index/our-decision-on-cursor-following-it... Its because of this. You can't use Astra in Cursor, and cursorbench uses cursor as the harness. They can't actually benchmark it using their harness hence why its not included.
- babelfish 14d ago
  They have Astra in other benchmarks lower on the page. They just don't want to show it winning
  - Jcampuzano2 14d ago
    The chart is cursorbench though and they asked about the "deceptive graph"
- ryeguy 14d ago
  They can benchmark it because you can use an openai api key with cursor. Astra is just not included in the cursor plan.
bluecalm 14d ago
Elon posted on X that Grok 4.7 is behind Claude and OpenAI for agentic coding: https://x.com/elonmusk/status/2102082011233931762?s=20 https://x.com/elonmusk/status/2102082011233931762?s=20 so it's likely about usage in Cursor specifically.

Jcampuzano2 14d ago

https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/ https://openai.com/index/our-decision-on-cursor-following-it... This explains why. Mentioned in another comment, but cursorbench explicitly tests with Cursor as the harness, and OpenAI doesn't allow them to use Astra in Cursor.

kristofferR 14d ago
That's not accurate. OpenAI doesn't allow Grok to provide Astra to Cursor customers anymore, but it doesn't ban anyone from using Astra via alternative harnesses. If Cursor wanted to include Astra in CursorBench nothing would stop them, they could easily have spent half an hour vibecoding in OpenAI API key support - if it hadn't been convenient to neglect to do that.
- andsoitis 14d ago
  Even if they could do that (workaround to include Astra in CursorBench), that has no practical consequences for Cursor users and that's what I as a Cursor user (what I use for dev, though I use ChatGPT for non-dev stuff) care about.
  - kristofferR 14d ago
    It would make the benchmark way better obviously, by showing how their new model compares to their competitors, the whole point of benchmarks and graphs.
    - Jcampuzano2 14d ago
      The point of Cursor Bench is to show how models perform in Cursor. If 99% of their users won't be able to access a model unless they go out of their way to include setup an API key for it (which would be insanely expensive with Astra), why would they include it in the benchmark?
user43928 14d ago

scottyah 14d ago

Deceptive? An extremely quick google search would answer your question. OpenAI pulled out of Cursor before they released Astra so it never got that benchmark.

kristofferR 14d ago
Pulled out from letting them resell Astra access, that's not a limitation on running a benchmark.