3 ms·
A lot of the crazy ideas seem to have melted away in the face of massive context sizes. Today, I can put roughly a megabyte of utf8 text into my system prompt b
by bob1029 4mo ago
A lot of the crazy ideas seem to have melted away in the face of massive context sizes. Today, I can put roughly a megabyte of utf8 text into my system prompt before things start to get weird.
That is a massive amount of information even if we are being sloppy with it. You can read The Hobbit and the first Harry Potter book cover-to-cover and still have room to spare. I would deeply struggle to develop a world model this detailed for any business. Anything that needs to get more specific than these narratives can be a SQL query tool into the data warehouse, grep over the codebase, MS graph API lookup, etc.
Giving the business a balanced way to collaborate over this one shared model of the world is a new challenge I am beginning to engage with. I've also noticed that the world model will compound on itself in terms of self-detection of update opportunities. The more constraints there are, the more likely we appear to violate one.
- stingraycharles 4mo agoYou’re forgetting that keeping the context small is better economically and delivers better results.
- themgt 4mo agoForgetting? I think you mean to say your advice was auto-compacted to keep our context small and deliver better results.
- bob1029 4mo agoI learned some nuance. Small context is important if the context isn't cachable. If most of it is the same prefix every time, the situation changes pretty significantly.
- embedding-shape 4mo ago> melted away in the face of massive context sizes If only. There is a huge difference between "Gives good responses/can easily spot things within N context size" and "Technically works but sucks within N context size", almost all models basically become cave-people once you go beyond 50% of the "supported" context size, meaning while they may technically work with 1 million output tokens, those last 500K tokens are gonna be massively "dumber" than the first 500k tokens.
- fooster 4mo agoI don’t find that to be true?
- ballsac 4mo ago[dead]
- stalfie 4mo agoThere's at least one benchmark that attempts to measure this, but it has been running for a year plus so it's quite infrequently updated now. https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...
- VerityLayer 4mo ago[flagged]