3 ms·
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the origi
by siva7 17d ago
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
- theLiminator 17d agoI wish someone ran some sort of representative benchmark suite every X days to see if this occurs.
- micycle1 17d agohttps://marginlab.ai/trackers/codex/ https://marginlab.ai/trackers/codex/
- CamperBob2 17d agoUnfortunately that seems to be monitoring Sol, not Astra, unless I'm missing something.
- embedding-shape 17d agoI'm not sure how you could run such a benchmark without leaving it possible for the labs to easily detect and fudge the results.
- nickreese 17d agoI had the same experience. Moving back to Sol for actual implementation.
- jcmontx 17d agoSame story every time, I bet they quantized it
- manmal 17d agoExactly my thoughts today. They have to make it cheaper after demoing what’s possible initially.
- cainxinth 17d agoIt's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
- blurbleblurble 17d agoOr a lot worse
- holler 17d agoso, AGI is cancelled?
- __MatrixMan__ 17d agoAGI for the peasants is cancelled.
- elwell 17d agoTrogdor - the AGInator
- baby 17d agoYou think they introduce stronger quantization after a few days?
- boredatoms 17d agoFor sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain
- NineStarPoint 17d agoYeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.
- ModernMech 17d agolol I didn't get access until Monday (I was at 0% since Friday and my reset was Sunday at 11pm), so go figure.
- scrlk 17d agoMight be related to this announcement from Tibo on Sunday: > We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account. > No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription. https://x.com/thsottiaux/status/2096717905614524491 https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491 https://xcancel.com/thsottiaux/status/2096717905614524491)
- siva7 17d agoIt seems to me the people working at OAI may believe all other humans must be a little bit behind intellectually.
- pixl97 17d agoI mean, in general they aren't wrong. You can't fool everybody all of the time, but you can fool almost everybody most of the time. But most of all, it's easy to fool yourself.
- rowanG077 17d agoIt always reminds of the story of the creator of counter strike. Every new release he would get a ton of complaints from players about things they didn't even change. Notably that each version had more lag. And he got so fed that he start to negatively subtract peoples pings. And suddenly a ton of players reported back that the change was incredibly good. Point is, I really don't buy all the stories about a model suddenly being downgraded without at least a modicum of substance. People are grasping at straws in the noise.
- Razengan 17d agoWhich plan/region are you on/in?
- siva7 17d agoHighest subscription tier and i believe there is only US region available being served globally
- mccoyb 17d agoI had nearly the exact same experience and thought I was imagining it … absolutely ripping, then it turned into Sol++ on Tuesday … I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not I can tell by analyzing my own prompts to look at when I get frustrated ;)
- binary0010 17d agoDisagree completely. I started using Astra from Sol the day it was released, and was a virtually imperceptable difference and made lots of mistakes and shit architecture decisions from day 1 of release.
- aetherspawn 16d agoI find Astra to be weirdly stupid in the sense you have to force it to spend time on something (fix this architectural issue and refactor), then it’s stupidly smart. It prioritises getting something working over making something good during the 1-shot phase and outputs maximum slop.
- kloop 16d agoI still think this is because, on a new release, it works on some prompts the previous ones did badly at, because new weights do well on a different set of prompts. Then after a few days you notice the prompts that it does badly on that the old ones did fine with and everyone is convinced there's a regression when it's just a different part of prompt space
- sobellian 17d agoI thought the same, but on second thought I merely had to deal on Tuesday with a lot of the mistakes Astra made on the preceding days. I wonder if this time lag of consequences explains why the sentiment is so common with these models. It probably also cautions against irrational exuberance when you first crack open a new model and it one-shots various problems, as you don't yet know what goats Astra had to sacrifice to make it so.
- throwatdem12311 17d agoI’m so used to seeing this on every single model release I’m starting to question if these kinds of posts are just trolling. Alternative theory - it always seems amazing when it first comes out then the novelty wears off and we’re just meh about it. New model is a model is a model. I bought a PS5 Pro and was genuinely blown away by it at first…few weeks later I’m just like…eh it looks pretty good I guess? It’s still the same, I’m just used to it now and the wow factor along a new thing is going. Kinda like that. Or they are just compute constrained so they have to serve a shittier version. Who knows? I hate how opaque these companies are. It feels deceptive and evil.
- acedTrex 17d agoThis shit is just vibe coder astrology lol
- konart 17d ago>now it feels like Sol It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.
- kgeist 16d ago>What stops them <..> simply use cheaper model for every Nth request. That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra. To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync. If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.
- qaq 17d agoOK so it's not just me
- Paracompact 16d agoCan you re-run some prompts that you ran on Monday and report the differences in output?
- dudeinhawaii 16d agoI have not experienced this (yet) but I have with models in the past. I think it's important to have a solid benchmark where you KNOW there's a difference in model performance. I have one around 3D modeling that models really land in the same space each time I run it. It's visual, and it's super clear. Sol has perpetually generated low quality work regardless of reasoning level. Astra was the first OpenAI model to suddenly leapfrog the pack and generate content that was production ready, beating out any other provider. I haven't seen Astra regress (yet). I think, if you want to be consistent and scientific about it, then you'd have to use the models via API and lock to a specific version. Via the subscriptions, you are floating on whatever the latest version is, vendor to vendor.