11 ms·
Grok 4 Fast now has 2M context window
- changoplatanero 1y agoAnyone can make a long context window. The key is if your model can make effective use of it or not.
- bigyabai 1y agoLong context window = huge amounts of vacant VRAM = our servers are fucking empty
- trash_cat 1y agoBut isn't context window dependent on model architecture and not available VRAM that you can just increase or decrease as you like?
- reasonableklout 1y agoMost attention implementations can work across an arbitrarily long context. The limiting factors are typically: 1. Often there are latency/throughput requirements for model serving which become challenging to fulfill at a certain context length. 2. The model has to be _trained_ to use the desired context length, and training becomes prohibitively expensive at larger contexts. (2) is even a big enough problem that some popular open source models that claim to support large context lengths in fact are trained on smaller ones and use "context length extension" hacks like YaRN to trick the model into working on longer contexts at inference time.
- onion2k 1y agoThe model will use the full context if it's been designed well, but you can still increase the size of the window on models where it hasn't. It's just pointless. People who don't know much about LLMs will still think "bigger number is better" though.
- chucknthem 1y agoHow do they make the context window longer? (serious question, I want to learn how this works)
- TheCoolGuy 1y agoYou literally just shift the window over by to the next token once you reach the max amount of tokens you want for context window, NOT with what you train on, (only limited with memory now) This has obvious issues since you're now losing information from the now unseen tokens which becomes significant if your context window is small in comparision of the answer/question you're looking at. That's why companies try to give stupidly large context windows. The problem is they're not training on the large context window, they're training on something smaller (2048 and above). Due to how attention is setup, you can train on a small amount of context and extrapolate it to any number of tokens possible since they train via ROPE which trains the model because on words and their offset to the neighboring words. This allows us to effectively x2,x3,x10,x100 the amount of tokens we generate vs train with with some form consistency BUT still cause a lot of issues consistency wise since the model approaches more of a "this was trained on snippets but not the entire thing" situation where it has a notion of the context but not fundamentally the entire combined context
- vlovich123 1y agoThat’s a very basic way to keep the LLM inferring past the context window size (there’s better, smarter ways) but that’s not at all what the question was which is how they train a 2M token length window. My understanding at a basic level is that you need corpuses that are >2M in length for training data which is where the problem comes in for - there’s only so much long form content and it’s swamped by all the smaller stuff. I think there’s probably tricks now but I suspect it’s still largely an open problem.
- Ey7NFZ3P0nzAe 1y agoAFAIK nobody does that. They train on much much shorter text but with use tricks in the position encoding steps that can be extrapolated by the LLMs. Lile ROPE and YARN etc.
- nbardy 1y agoNo they can't, it's a N^2 algorithm, just fitting it in the context window is a challenge. And sure maybe not 2mil of it is usable, but they're reliably pushing the frontier here.
- ggeorgovassilis 1y agoI came here just to complain about that :-) All LLMs I used seem to give more weight to things at the beginning of the context window and omit many details. Eg. I tried this simple thing: pasted a friend's and my CV into Gemini and asked it to recommend topics for a joint conference presentation. Results depended greatly on the order of CVs pasted in.
- TheOtherHobbes 1y agoThe middle tends to be underweighted. The beginning and end get more attention.
- otabdeveloper4 1y agoThat's because when they say "long context window" they're lying and they actually mean that they support a long input prompt that is still compressed into a small context window. (Typically by throwing out tokens in the middle.) An actually large context window is impossible due to how LLM attention works under the hood.
- acuozzo 1y agoMamba-2 enters the chat.
- retinaros 1y agono one makes effective use of long context.
- DrSiemer 1y agoIt's not the most energy efficient workflow, but I work on relatively small codebases and I made a tool that let's me dump all of it in an LLM with a single copy/paste. This works surprisingly well with Gemini 2.5 Pro (1.000.000 ctx). The only real mistakes it makes are some model specific quirks, like occasionally stripping out certain array index operators. Other than that, it works fine with 150.000 token size conversations. I've gone up to 500.000 with no real issues besides a bit of a slowdown. It's also great for log analysis, which I have maximized to 900.000 tokens.
- mg 1y agoIf a model is not making use of the whole context window - shouldn't that be very noticeable when the prompt is code? For example when querying a model to refactor a piece of code - would that really work if it forgets about one part of the code while it refactors another part? I concatenate a lot of code files into a single prompt multiple times a day and ask LLMs to refactor them, implement features or review the code. So far, I never had the impression that filling the context window with a lot of code causes problems. I also use very long lists of instructions on code style on top of my prompts. And the LLMs seem to be able to follow all of them just fine.
- MallocVoidstar 1y agoI don't think there are any up-to-date leaderboards, but models absolutely degrade in performance the more context they're dealing with. https://wandb.ai/byyoung3/ruler_eval/reports/How-to-evaluate-the-true-context-length-of-your-LLM-using-RULER---VmlldzoxNDE0OTA0OQ https://wandb.ai/byyoung3/ruler_eval/reports/How-to-evaluate... >Gpt-5-mini records 0.87 overall judge accuracy at 4k [context] and falls to 0.59 at 128k. And Llama 4 Scout claimed a 10 million token context window but in practice its performance on query tasks drops below 20% accuracy by 32k tokens.
- mg 1y agoThat makes me wonder if we could simply test this by letting the LLM add or multiply a long list of numbers? Here is an experiment: https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%20number.%20Do%20not%20use%20a%20calculator.%20Do%20it%20in%20your%20head.%0A%0A(1.1%20%2B%202.5)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1)%20*%20(5%20%2B%201)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1) https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%... The correct answer: Correct: 20,192,642.460942328 Here is what I got from different models on the first try: ChatGPT: 20,384,918.24 Perplexity: 20,000,000 Google: 25,167,098.4 Mistral: 200,000,000 Grok: Timed out after 300s of thinking
- 1y ago
- d4rkp4ttern 1y agoThere are “needle in the haystack” benchmarks for long context performance. It would be good to see those.
- throwuxiytayq 1y agoThese aren’t really indicative of real world performance. Retrieving a single fact is pretty much the simplest possible task for a long context model. Real world use cases require considering many facts at the same time while ignoring others, all the while avoiding the overall performance degradation that current models seem susceptible to when the context is sufficiently full.
- d4rkp4ttern 1y agoI agree, retrieving a single fact is necessary but not sufficient.
- deleted 1y ago[deleted]
- jtrn 1y agoThe number of times I know that my instruction is in context, but it’s forgotten, is countless at this point for me. My experience, both ad a clinical psychologist and developers, is that there is a convergent trend in how I speak to both clients and AI. I can view much of my therapist's approach in how I try to highlight the important things to focus on to achieve progress. Often, it’s about helping the client articulate and understand what’s important to them and how they rank these priorities. The same applies to AI. It feels obvious now that the problem with attention and context is the lack of hierarchy or levels of importance. We know that we have, probably biologically based, three types of memory: short-term, intermediate, and long-term. Long-term memory is what you use with MCP, web search, and RAG. Shorter memory is the current response, and intermediate memory is the current context. When assume this, in my interactions with an agent, it makes perfect sense where they falter and what they forget, in the exact same way as people. It feels more and more like talking to a human, with same weaknesses in logic, reasoning, and focus.
- behnamoh 1y agoWho here actually uses Grok? It's sad to see Elon's arc but when he doubled down on some of his political ideas he had it coming with the Tesla sales going down and x.ai not taken seriously. I've always tried to remain apolitical and unbiased but it's hard to overlook who's behind a technology you wanna buy. Not that sama and others are saints either, it's just Elon's very obvious and vocal about it. It's a shame, really, because Grok is a good model. But Elon promised to open source the previous model and it took them forever to do that with Grok 3. Sorry, but I wanna buy from someone who keeps their promises ("FSD by next year").
- wetpaws 1y ago[dead]
- jacquesm 1y ago[flagged]
- YetAnotherNick 1y agoGrok fast is by far the most used model in openrouter with more than a trillion tokens weekly[1]. [1]: https://openrouter.ai/rankings https://openrouter.ai/rankings
- behnamoh 1y agoBecause some tools (AFAIR Kilo Code but I might be wrong) gave it away for free. The model itself was (still is?) free for a while, so I'm not surprised.
- ribelo 1y agoOpenrouter is not counting tokens used by Kilo or Cline. They have own endpoints.
- wqaatwt 1y agoYet if you go to the actual model’s page: https://openrouter.ai/x-ai/grok-code-fast-1 https://openrouter.ai/x-ai/grok-code-fast-1 Cline and Kilo code are in the top 3. So how does that work? It’s considerably cheaper than competing models like 2.5 flash, though. So its not that surprising
- mehdibl 1y agoWhat matter is not context or the recod token/s you get. But the quality for the model. And it seem Grok pushing the wrong metrics again, after launching fast.
- saretup 1y agoSeems reductive. Some applications require higher context length or fast tokens/s. Consider it a multidimensional Pareto frontier you can optimize for.
- sigmoid10 1y agoIt's not just that some absolutely require it, but a lot of applications hugely benefit from more context. A large part of LLM engineering for real world problems revolves around structuring the context and selectively providing the information needed while filtering out unneeded stuff. If you can just dump data into it without preprocessing, it saves a huge amount of development time.
- cronin101 1y agoDepending on the application, I think “without preprocessing” is a huge assumption here. LLMs typically do a terrible job of weighting poor quality context vs high quality context and filling an XL context with unstructured junk and expecting it to solve this for you is unlikely to end well. In my own experience you quickly run into jarring tangents or “ghosts” of unrelated ideas that start to shape the main thread of consciousness and resist steering attempts.
- sigmoid10 1y agoIt depends to the extent I already mentioned, but in the end more context always wins in my experience. If you for example want to provide a technical assistant, it works much better if you can provide an entire set of service manuals to the context instead of trying to put together relevant pieces via RAG.
- 1y ago
- cactusplant7374 1y agoI had a failed refactor with Codex recently and I am wondering if context window size is the cause.
- sgc 1y agoI not an expert ai user (and have never touched Codex), but anything remotely important I do, I force the smallest context window possible. I just did something very beautiful using that principle, which will soon be ready to show the world. It would have been a garbled pile of garbage with long context windows. Obviously major architectural changes need a bigger context window. But try to aggressively modularize your tasks as much as you can, and where possible run batch jobs to keep your workflow moving while each task stays a smaller chunk.
- jakevoytko 1y agoWith the current crop of LLMs/agents, I find that refactors still have to be done at a granular level. "I want to make X change. Give me the plan and do not implement it yet. Do the first thing. Do the second thing. Now update the first call site to use the new pattern. You did it wrong and I fixed it in an editor; update the second call site to match the final implementation in $file. Now do the next one. Do the next one. Continue. Continue.", etc.
- enraged_camel 1y agoFor complex refactors, I use "max mode" in Cursor, which in my experience noticeably improves the AI's performance and makes it go for a lot longer before it starts to drift. I haven't looked into how it works exactly, but it works well if you don't mind the extra cost.
- whywhywhywhy 1y agoHad some bad experiences with max mode and the latest Claude spending significant time on writing worthless .md files rather than solving problems
- port3000 1y agoI use Claude Code, haven't used Codex yet (should I?) - but in Claude code you can spin up sub-agents to handle these big refactors, with the master context window just keeping track of the overall progress, bugs, etc and providing instructions to the subagents to do the rote work.
- johnnyApplePRNG 1y agoBut for some reason if I load a 400kb file into it... it can't even read the file?! Pffft, whatever elon. Go play with your rockets.
- straydusk 1y ago[flagged]
- Lapel2742 1y ago[flagged]
- IshKebab 1y agoCome on, I would also never use Grok because Elon is an arsehole. He's not a neonazi though. That's ridiculous.
- jofzar 1y agoDoes the internet move so fast that people have forgotten his Roman salute at the trump rally?
- wqaatwt 1y agoThat proves he’s an edgy, drugged out jerk. Raising your hand in a certain way is not sufficient to be a nazi..
- Lapel2742 1y ago[flagged]
- wqaatwt 1y agoSo let’s focus on the specifics things he does and says. Instead of compressing everything into a single word (which unfortunately lost much of its meaning and has been massively devalued). Nazism is a specific, relatively, defined and extremely dark ideology. If we apply it to every unhinged off brand pseudo-fascist it can really distort the views people might start having about the original ideology.
- raincole 1y agoIt's funny how fast this post is flagged, lol. Have other LLMs or blunt ads got the same treatment on HN?
- hereme888 1y ago[flagged]
- jauntywundrkind 1y ago[flagged]
- j3th9n 1y agoIn what universe do you live.
- jauntywundrkind 1y agoI believe those people are eager to discuss Musk. The people suppressing Musk discussion are the forces backing him, who are out here working to suppress inconvenient speakings.
- hereme888 1y agoI honestly can't compute the logic of your statement.
- latexr 1y ago> Have other LLMs or blunt ads got the same treatment on HN? Yes, I’ve seen it happen multiple times.
- ronsor 1y agoThis post really has no reason to be flagged. I know Elon is controversial, and I have a lot of gripes with his business practices myself, but this is literally just documentation for a frontier LLM. Can we stay on topic?
- big-and-small 1y agoThis. I wouldn't pay to use it, but big context windows are amazing for programming and especially prototyping when you can keep whole codebase in context. Gemini's 1M is amazing.
- ergocoder 1y ago[flagged]
- poly2it 1y agoAre you implying that avoiding use of services controlled by a fascistoid oligarch is cope?
- ergocoder 1y agoNot really. If we are gonna avoid using the services by an oligarch, then just say that. No need to go through the 5 steps of denial. I hope we are not against being honest.... but you can see a bunch of comments here going through those steps already.
- oulipo2 1y agoThe politics of the owners IS the topic. It's being really naive (read: stupid) to think that this has no implication on society
- TheOtherHobbes 1y agoYou're literally handing over your code to a third party. In fact AI is handing over the process of creating code - eventually all code - to a small number of third parties, who will have complete power over the world's IT infrastructure. No wonder they have wildly inflated valuations. The potential to enforce authoritarian policies through opaque technology is unprecedented.
- ml-anon 1y ago[flagged]
- ribelo 1y agoI give a shit, and I use it every fucking day.
- oulipo2 1y ago[flagged]
- 458632058268 1y ago[flagged]
- nsoonhui 1y agoIt's a shame that the top comments are focusing more on Elon Musk, his personality and politics rather than the quality of the model per se. Speaking about Elon, regardless of what you think of him, he really does get things done, despite naysayers -- SpaceX, Tesla, Neuralink and even get Trump elected ( despite subsequent fallout) etc. Even Twitter is finding a second life by becoming a haven for the free speech advocates and alternative views, much to the chagrin of MSMs because they now no longer have the monopoly on the "truth", and censoring "fake news" becomes hard. People like Elon are almost by definition contrarian ( you don't change the world by being a conformist), that should align well with the predilection of the intended audience here. So it's a surprise to me that HNs are almost uniformly, vehemently anti-Musk. It's almost as if the ultimate embodiment of the hacker spirit -- Musk -- is being rejected by his own kind, the very kind that he is supposed to inspire.
- letmetweakit 1y agoIn my understanding of the hacker ethos, hackers appear to be genuinely nice people who mean to do good for society and regular people. Elon does not align with those values according to some people so they reject him and his activities.
- vachina 1y ago[flagged]
- victorbjorklund 1y agoAccusing a cave diver who made Elon look stupid to be a pedophile just because Elon can’t stand people not thinking he is the smartest? Can give more examples.
- finebalance 1y agoKilling USAID.
- bertili 1y agoBesides the obvious right wing interference in politics, star link weaponization in some countries - how can anybody stomach the saving-humanity-agenda while running a major social media unresponsiveliy without caring of moderation, its consequences for real people?
- jeffhuys 1y ago[flagged]
- oulipo2 1y ago[flagged]
- jasonvorhe 1y agoWhat a grandiose nuanced statement. Thanks for this highly enlightening contribution!
- himujjal 1y agoI think this is against HN’s policies. @dang
- angusturner 1y agoI thought exceptions tended to be made when its highly relevant to the technical topic at hand and also non controversial. Outside a few weird online bubbles and pockets of the US, hardly anyone disputes the claim you are objecting to.
- hu3 1y agoRegardless, it's just noise.
- oulipo2 1y agoIf you don't understand that politics, ideology and technology are deeply intertwined and they simply CANNOT be considered separately, you should start educating yourself
- angusturner 1y ago[flagged]
- solumunus 1y agoGrok? Next…
- tacker2000 1y agoYea, no desire to ever use this.
- bushbaba 1y agoI personally find grok better for certain tasks. It’s better than Gemini for images. Its better than the rest at crude jokes etc
- drivingmenuts 1y agoHonestly, if Elon Musk told me what time it was, I wouldn't trust him.
- htrp 1y agoAny details on exactly how they accomplished this? longrope?
- daft_pink 1y agoMy experience with AI is that you generally want to keep your context as small as possible and this is only useful when your relevant context is actually 2m tokens.
- hereme888 1y agoThat's my experience as well.
- deleted 1y ago[deleted]
- Frannky 1y agoI started with ChatGPT, then moved on to Claude, and then discovered Grok. But now I've stopped paying for any of them. Claude edged out ChatGPT in quality, while Grok stood out with its generous usage limits. That all changed, though, once they rolled out the agent system and RLHF. Suddenly, the model slowed to a crawl, veering off on wrong paths and getting lost in its own reasoning. Those endless, super-annoying RLHF popups didn't help either. My theory? They were scrambling for a competitive edge and were willing to swallow some short-term pain. Plus, it feels like they shifted focus away from keeping coders deeply in the loop. In the end, we vote with our wallets—if it doesn't click, just walk away. I still dip into Grok, but only the free tier: Grok 4's fast mode for tackling planning and first generation, and then Qwen Coder for the code editing and clerical tasks. The latest version of grok hold up about as well as the old Grok 3, just with way more steps...
- giancarlostoro 1y agoI guess I joined Claude late, but its been working pretty decent for me. I've been using Claude Code with Zed now that it's a native feature. Honestly, if you're building coding APIs for your LLM and you aren't working with the Zed folks to get your model natively in that editor, you're messing up big in my eyes, its just done so well. My biggest gripe with Grok is they're not really integrated in all the great tooling I use. I know I can use an API key with Zed, but come on, you want to compete with something like Claude Code? You need to integrate with the tools devs actually use. If they want to rush on anything, get it on more tools.
- Frannky 11mo agoI complained to them about the missing CLI. That was probably the last straw that made me decide to stop paying for it. They could deliver a CLI with calls included in SuperGrok, and a lot of people would stop using Gemini CLI and Qwen Code.
- vaxman 1y agoOpenAI will go to zero unless it agrees to be acquired because they're messing with public company stock valuations using funky purchase orders leaving those public companies no choice but to cancel their credit (at least unless they get a "government backstop" that they say they don't want or need). Those who compete with OpenAI will also "take a hit" if/when that happens, so they would be wise to be looking to make a deal to acquire OpenAI. Dude was from Y Combinator and liked to bank on hope, focusing on capturing market share and worrying about profits later, which is fine in software startups playing with Monopoly money, but when it impacts vendors that are publicly traded companies (to the point that one is now valued at $5T), post-1929 rules come into play. Anthropic has a similar issue, but there, the issue is that their C-suite is making outrageous public statements that are suspected of intending to manipulate the stock values of both private and public competitors and of the publicly held vendors to all of these players. I hope they both go away quietly and someone declares victory rather than the stock market crashing! As far as xAI, I doubt it will go to zero or run afoul of any of those market manipulation issues because it owns Twitter/X and I think it powers the realtime Tesla cloud, but betting on it is fraught with peril because of the high likelihood that it will wind up under the control of some less capable conglomerate (ergo, GM acquisition of Hughes Aircraft and resale to Raytheon, Boeing and News/DirecTV). Google, Meta, a handful of B actors and China are where we have to place our bets, but only if we ourselves need (or want to invest on the theory that others need) trillion parameter models (and want to risk having the valuations lowered if/when adverse actions are taken against the above competitors).
- vaxman 1y ago*-"eventually" leaving those public companies no choice but to... Clarifying, because there's no way a company (public or private) is going to reduce the credit line of a major customer until it's obvious that the orders "aren't real" But if Wall Street realizes it before they do, they can lose control of their business too. This is not quite Enron or WorldCom/MFS, but it's a very similar storm on the horizon. (BTW, ever wonder why Sprint never could remain airborne and eventually was merged with TeenMobile? It's because they overspent on CapX trying to keep up with the fraud at Worldcom and could never dig out to actually use all that spectrum. Likewise, we are still dealing with the fallout of the Enron collapse on the US domestic energy grid a quarter century later.)