11 ms·
> Claude 3 was the first model where we added "character training" to our alignment finetuning process: the part of training that occurs after initial model tra
by simonw 2y ago
> Claude 3 was the first model where we added "character training" to our alignment finetuning process: the part of training that occurs after initial model training, and the part that turns it from a predictive text model into an AI assistant. The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness.
What I found particularly interesting is how they implemented this using primarily synthetic data:
> We ask Claude to generate a variety of human messages that are relevant to a character trait—for example, questions about values or questions about Claude itself. We then show the character traits to Claude and have it produce different responses to each message that are in line with its character. Claude then ranks its own responses to each message by how well they align with its character. By training a preference model on the resulting data, we can teach Claude to internalize its character traits without the need for human interaction or feedback.
- yard2010 2y ago"Open mindedness - is that even a word?"
- jraph 2y agoIt seems so. The Wiktionary entry [1] has existed since 2007 and lists translations to several languages. The French one is quite common, it makes sense that English has a counterpart. https://en.wiktionary.org/wiki/open-mindedness https://en.wiktionary.org/wiki/open-mindedness
- cobbzilla 2y agohave you considered that you may hold the opposite attribute?
- idiotsecant 2y agoYou've never heard that phrase? Its pretty common.
- wongarsu 2y agoWhy would it not be? You can form "open-mindedness" from "open minded" the same you can form "yellowness" from "yellow" or "coldness" from "cold". It's a valid way to make a word from almost any adjective. Syntactically it's weird because unlike most other Germanic languages English likes to put spaces into compound words. But that hasn't stopped open-mindedness from catching on and getting mentioned in dictionaries.
- shlubbert 2y agoI suppose the audience here never watched The Office.
- TeMPOraL 2y agoIt's English. Any verb can be nouned, and you can verb any noun, and people will understand you just fine. It's a feature, perhaps more cultural than linguistic, but a feature nonetheless.
- HarHarVeryFunny 2y ago> The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness. You can't make a transformer-based language model "have" curiosity. Real curiosity is an innate trait in intelligent animals that promotes learning and exploration by increasing focus and stick-to-it-ness when the situation is interesting/unexplored (i.e. not well predicted - surprising). An LLM could be trained to fake curiosity in same way that ELIZA did ("Tell me more about your fear of prison showers"), but fundamentally it deals in words and statistics, not facts and memories, so any poorly-predicted surprise signal will be surface level such as "that was an unusual word pattern" or "you haven't mentioned that before in the input context". Even if fake curiosity ("interest") makes the model seem more conversationally engaging, maybe helps exploration by prompting the use to delve deeper into a subject, it's not going to help learning since LLMs are pre-trained and cannot learn by experience (except ephemerally within context).
- mckirk 2y agoI think you _can_ make an LLM 'have' curiosity, for all practical intents and purposes. I'm e.g. thinking of the 'have the LLM research a topic' task, such as the 'come up with suitable search terms, search the web, summarize the article, then think of potential next questions' cycle implemented by Perplexity, for example. I'm pretty sure the results would vary noticeably between an LLM that was trained to be 'curious' (i.e., follow more unusual trains of thought) versus not, and the differences would probably compound the more freedom you give the LLM, for example by giving it more iterations of the 'formulate questions, search, summarize' loop.
- HarHarVeryFunny 2y agoThe problem is how can "follow more unusual trains of thought" apply to a language model ? Sure it can selectively attend to certain parts of the input and generate based on that, but what is the internal signal for "unusual" ? Any selective focus is also going to appear like groundhog day since the model's weights are fixed and what was "unusual" today will continue to be "unusual" even after it's been exposed to it for the 1000th time!