7 ms·
“Time-locked models don't roleplay; they embody their training data. Ranke-4B-1913 doesn't know about WWI because WWI hasn't happened in its textual universe. I
by saaaaaam 10mo ago
“Time-locked models don't roleplay; they embody their training data. Ranke-4B-1913 doesn't know about WWI because WWI hasn't happened in its textual universe. It can be surprised by your questions in ways modern LLMs cannot.”
“Modern LLMs suffer from hindsight contamination. GPT-5 knows how the story ends—WWI, the League's failure, the Spanish flu.”
This is really fascinating. As someone who reads a lot of history and historical fiction I think this is really intriguing. Imagine having a conversation with someone genuinely from the period, where they don’t know the “end of the story”.
- observationist 10mo agoThis is definitely fascinating - being able to do AI brain surgery, and selectively tuning its knowledge and priors, you'd be able to create awesome and terrifying simulations.
- eek2121 10mo agoRespectfully, LLMs are nothing like a brain, and I discourage comparisons between the two, because beyond a complete difference in the way they operate, a brain can innovate, and as of this moment, an LLM cannot because it relies on previously available information. LLMs are just seemingly intelligent autocomplete engines, and until they figure a way to stop the hallucinations, they aren't great either. Every piece of code a developer churns out using LLMs will be built from previous code that other developers have written (including both strengths and weaknesses, btw). Every paragraph you ask it to write in a summary? Same. Every single other problem? Same. Ask it to generate a summary of a document? Don't trust it here either. [Note, expect cyber-attacks later on regarding this scenario, it is beginning to happen -- documents made intentionally obtuse to fool an LLM into hallucinating about the document, which leads to someone signing a contract, conning the person out of millions]. If you ask an LLM to solve something no human has, you'll get a fabrication, which has fooled quite a few folks and caused them to jeopardize their career (lawyers, etc) which is why I am posting this.
- libraryofbabel 10mo agoThis is the 2023 take on LLMs. It still gets repeated a lot. But it doesn’t really hold up anymore - it’s more complicated than that. Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. Sure, LLMs do not think like humans and they may not have human-level creativity. Sometimes they hallucinate. But they can absolutely solve new problems that aren’t in their training set, e.g. some rather difficult problems on the last Mathematical Olympiad. They don’t just regurgitate remixes of their training data. If you don’t believe this, you really need to spend more time with the latest SotA models like Opus 4.5 or Gemini 3. Nontrivial emergent behavior is a thing. It will only get more impressive. That doesn’t make LLMs like humans (and we shouldn’t anthropomorphize them) but they are not “autocomplete on steroids” anymore either.
- deadbolt 10mo agoAs someone who still might have a '2023 take on LLMs', even though I use them often at work, where would you recommend I look to learn more about what a '2025 LLM' is, and how they operate differently?
- krackers 10mo agoPapers on mechanistic interpratability and representation engineering, e.g. from Anthropic would be a good start.
- otabdeveloper4 10mo agoDon't bother. This bubble will pop in two years, you don't want to look back on your old comments in shame in three.
- dingnuts 10mo ago[dead]
- root_axis 10mo ago> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By refining training data quality, increasing parameter size, and increasing context length we can squeeze more utility out of LLMs than ever before, but ultimately, Opus 4.5 is the same thing as GPT2, it's only that coherence lasts a few pages rather than a few sentences.
- ilaksh 10mo agoActivation steering can do that to some degree, although normally it's just one or two specific things or rather than a whole set of knowledge.
- nottorp 10mo agoYou can't. To use your terms, you have to "grow" a new LLM. "Brain surgery" would be modifying an existing model and that's exactly what they're trying to avoid.
- xg15 10mo ago"...what do you mean, 'World War One?'"
- tejohnso 10mo agoI remember reading a children's book when I was young and the fact that people used the phrase "World War One" rather than "The Great War" was a clue to the reader that events were taking place in a certain time period. Never forgot that for some reason. I failed to catch the clue, btw.
- bradfitz 10mo agoI seem to recall reading that as a kid too, but I can't find it now. I keep finding references to "Encyclopedia Brown, Boy Detective" about a Civil War sword being fake (instead of a Great War one), but with the same plot I'd remembered.
- michaericalribo 10mo agoCan confirm, it was an Encyclopedia Brown book and it was World War One vs the Great War that gave away the sword as a counterfeit!
- JuniperMesos 10mo agoThe Encyclopedia Brown story I remember reading as a kid involved a Civil War era sword with an inscription saying it was given on the occasion of the First Battle of Bull Run. The clues that the sword was a modern fake were the phrasing "First Battle of Bull Run", but also that the sword was gifted on the Confederate side, and the Confederates would've called the battle "Manassas Junction". The wikipedia article https://en.wikipedia.org/wiki/First_Battle_of_Bull_Run https://en.wikipedia.org/wiki/First_Battle_of_Bull_Run says the Confederate name was "First Manassas" (I might be misremembering exactly what this book I read as a child said). Also I'm pretty sure it was specifically "Encyclopedia Brown Solves Them All" that this mystery appeared in. If someone has a copy of the book or cares to dig it up, they could confirm my memory.
- 10mo ago
- jscyc 10mo agoWhen you put it that way it reminds me of the Severn/Keats character in the Hyperion Cantos. Far-future AIs reconstruct historical figures from their writings in an attempt to gain philosophical insights.
- bikeshaving 10mo agoThis isn’t science fiction anymore. CIA is using chatbot simulations of world leaders to inform analysts. https://archive.ph/9KxkJ https://archive.ph/9KxkJ
- NuclearPM 10mo ago[flagged]
- A4ET8a8uTh0_v2 10mo agoInteresting. Would you be ok disclosing the following: - Are you ( edit: on a ) paid version? - If paid, which model you used? - Can you share exact prompt? I am genuinely asking for myself. I have never received an answer this direct, but I accept there is a level of variability.
- ghurtado 10mo agoDepending on which prompt you used, and the training cutoff, this could be anywhere from completely unremarkable to somewhat interesting.
- BoredPositron 10mo agoI call bullshit because of tone and grammar. Share the chat.
- DonHopkins 10mo agoOnce there was Fake News. Now there is Fake ChatGPT.
- 10mo ago
- culi 10mo agoI used to follow this blog — I believe it was somehow associated with Slate Star Codex? — anyways, I remember the author used to do these experiments on themselves where they spent a week or two only reading newspapers/media from a specific point in time and then wrote a blog about their experiences/takeaways On that same note, there was this great YouTube series called The Great War. It spanned from 2014-2018 (100 years after WW1) and followed WW1 developments week by week.
- tyre 10mo agoThe Great War series is phenomenal. A truly impressive project.
- verve_rat 10mo agoThe people that did the Great War series (at least some of them, I believe there was a little bit of a falling out) went on to do a WWII version on the World War II channel: https://youtube.com/@worldwartwo https://youtube.com/@worldwartwo They are currently in the middle of a Korean War version: https://youtube.com/@thekoreanwarbyindyneidell https://youtube.com/@thekoreanwarbyindyneidell
- rcpt 10mo agoWatching a modern LLM chat with this would be fun.
- Sieyk 10mo agoI was going to say the same thing. Its really hard to explain the concept of "convincing but undoubtedly pretending", yet they captured that concept so beautifully here.
- ghurtado 10mo agoThis might just be the closest we get to a time machine for some time. Or maybe ever. Every "King Arthur travels to the year 2000" kinda script is now something that writes itself. > Imagine having a conversation with someone genuinely from the period, Imagine not just someone, but Aristotle or Leonardo or Kant!
- RobotToaster 10mo agoI imagine King Arthur would say something like: Hwæt spricst þu be?
- yorwba 10mo agoWrong language. The Arthur of legend is a Celtic-speaking Briton fighting against the Germanic-speaking invaders. Old English developed from the language of his enemies. https://en.wikipedia.org/wiki/Celtic_language_decline_in_England https://en.wikipedia.org/wiki/Celtic_language_decline_in_Eng...
- anthk 10mo agoEasier with Cervantes for Spanish speakers than King Arhur or Shakespeare. With Alphonse X, o The Cid, it would be greater issues, but understandable over weeks.
- Davidbrcz 10mo agoThat's some Westworld level of discussion
- psychoslave 10mo ago>Imagine having a conversation with someone genuinely from the period, where they don’t know the “end of the story”. Isn't this part of the basics feature of human conditions? Not only we are all unaware of the coming historic outcome (though we can get some big points with more or less good guesses), but to a marginally variable extend, we are also very unaware of past and present history. LLM are not aware, but they can be trained on larger historical accounts than any human and regurgitate syntactically correct summary on any point within it. Very different kind of utterer.
- pwillia7 10mo agocaptain hindsight
- psychoslave 10mo agoActually, this made me discover the character, thanks. I see your point and get the fun out of myself. On the other hand, at least in this case I don't pretend to cover some catastrophic results. :)
- pwillia7 10mo agoThis is why the impersonation stuff is so interesting with LLMs -- If you ask chatGPT a question without a 'right' answer, and then tell it to embody someone you really want to ask that question to, you'll get a better answer with the impersonation. Now, is this the same phenomenon that causes people to lose their minds with the LLMs? Possibly. Is it really cool asking followup philosophy questions to the LLM Dalai Lama after reading his book? Yes.
- Sprotch 10mo agoNice idea, does not work
- pwillia7 10mo agoIn which way?
- staticman2 10mo agoWhy is that cool? Imagine you are a billionaire so money is no object and really interested in the Dhali Llama? Would you read the book then hire someone to pretend to be the author and ask questions that are not covered by the book? Then be enraptured by whatever the roleplayer invents? Probably not? At least this isn't a phenomenon I've heard of?
- anshumankmr 10mo ago>where they don’t know the “end of the story”. Applicable to us also, cause we do not know how the current story ends either, of the post pandemic world as we know it now.
- DGoettlich 10mo agoexactly
- ViktorRay 10mo agoReminds me of this scene from a Doctor Who episode https://youtu.be/eg4mcdhIsvU https://youtu.be/eg4mcdhIsvU I’m not a Doctor Who fan and haven’t seen the rest of the episode and I don’t even what this episode was about but I thought this scene was excellent.
- takeda 10mo ago> This is really fascinating. As someone who reads a lot of history and historical fiction I think this is really intriguing. Imagine having a conversation with someone genuinely from the period, where they don’t know the “end of the story”. Having the facts from the era is one thing, to make conclusions about things it doesn't know would require intelligence.
- dr-detroit 10mo ago[dead]
- LordDragonfang 10mo agoPerhaps I'm overly sensitive to this and terminally online, but that first quote reads as a textbook LLM-generated sentence. "<Thing> doesn't <action>, it <shallow description that's slightly off from how you would expect a human to choose>" Later parts of the readme (whole section of bullets enumerating what it is and what it isn't, another LLM favorite) make me more confident that significant parts of the readme is generated. I'm generally pro-AI, but if you spend hundreds of hours making a thing, I'd rather hear your explanation of it, not an LLM's.
- Sprotch 10mo agoThis is the point - a modern LLM "role playing" pre-1913 would only reflect our view today of what someone from that era would say. It woud not be accurate.
- diamond559 10mo agoYeah, whenever we figure out time travel that will be really cool. In the meantime we have autocorrect trained on internet facts and modern textbooks that can never truly understand anything let alone what is was like to live hundreds of years ago.
- throawayonthe 10mo agoi get what you're saying, but the post is specifically about models that were not trained on the internet/modern textbooks