11 ms·
A simulation of me: fine-tuning an LLM on 240k text messages
- wyldfire 3y ago> I immediately recognized this would be in extremely poor taste and removed the data. Enough on that topic! Poor taste for a blog post. But I wonder if LLMs could advance to a point where it might be therapeutic to have a conversation with a deceased loved one.
- eddonner 3y agoGreat point. Bound to be a big area of debate as LLMs get increasingly good at this. Black Mirror’s take: https://en.wikipedia.org/wiki/Be_Right_Back https://en.wikipedia.org/wiki/Be_Right_Back
- RobRivera 3y agoOh hey Black Mirror!
- V__ 3y agoCould probably be just as dangerous as helpful. You work through some issues, maybe want to forgive or be forgiven to get closure and the LLM says f*k you, no.
- jononor 3y agoThey will RLHF* that away, as best as they can. * Reinforcement Learning from Human Feedback
- mewpmewp2 3y ago> it might be therapeutic to have a conversation with a deceased loved one. Would it? I'm not a specialist, but it would seem a bit off to me to approach it like that. I'd think best is to accept that your deceased one has gone, and it seems like having a fake one might just prolong the grievance process and possibly also develop into an obsession of trying to replace your deceased one with the simulation, upgrading it and obsessively finding ways to make it more accurate. If you weren't able to process your loved one being gone in the first place, how would you ever be able to quit from using the simulation? Although there is a market for psychics who claim to be able to talk to deceased relatives. Despite them obviously being liars, I wonder what the effect overall is for the customers. Does it help them? And is it because they believe the psychic, e.g. if psychic says, that your loved one would want you to move on and live your best life, perhaps this could alleviate their guilt. But I'm not sure how that would play out with a simulation. Would it have to be simulation's goal to slowly wean them off while hinting them to move on with their life and live their best life. But if I was to find out I am about to die, I would definitely want to try and build a simulation out of myself, that would then keep DMing the people I knew. I guess either it would have to be activated as marked down in the will, or it would have to be a sort of dead man's switch. I have to reset it monthly or it will have a timer to activate. Once I die, people will keep receiving DMs of the simulated version.
- alchemist1e9 3y agoHave you ever watched the original Superman movies and Clark talks with the AI hologram of his biological father from his home planet? I believe in the future our descendants will likely find it normal to talk with AI versions of their ancestors, perhaps not us parents right now, but I suspect my children will willingly train and generate AI versions of themselves for future generations to interact with.
- jazzyjackson 3y agoI also get the vibe that digital cloning will be popular. Maybe some extremists will think its unholy and some addicts will lose sense of reality, but for the vast majority of people, I think it's just a user interface - maybe to some particular piece of information in the chatbot-as-librarian role, or puzzle boxes that eventually reveal information once you ask the right question, like Will Smith in I Robot. What I don't like is that the better it gets the more a radical contigent will start to believe that you're performing brain uploading, and people will want to protect these simulations as people with rights, or ban it to prevent conscious mini-mes from being booted into a simulation and psychologically tortured like the guy in Altered Carbon. Then of course theere's the suicide cults like in the Cowboy Bebop episode "Brain Scratch" - leave your aches and pains behind ! I pray it doesn't get that weird! But if I had kids I wouldn't be raising them to talk to ghosts!
- pixl97 3y ago> Maybe some extremists will think its unholy At the other side of this equation is the extremists that think this is great. To go full Godwin's law there are certainly people out there that would like for a digital Hitler to stick around forever.
- HKH2 3y agoI'm sure people will create holographic Hitlers for historical reasons too.
- 3y ago
- AriedK 3y agoIt has been tried. Joshua Barneau tried it with GPT-3, training it on his deceased fiancees text- and facebook messages. It’s a moving read: https://www.sfchronicle.com/projects/2021/jessica-simulation-artificial-intelligence/ https://www.sfchronicle.com/projects/2021/jessica-simulation... It ‘worked’ for him for a while.
- deleted 3y ago[deleted]
- lomir8 3y agoShocking a biochemical reaction ceases when external stimuli is removed. Technology is like acupuncture and astrology of the modern west.
- ben_w 3y agoUnless there is an acupuncture powered smartphone and an astrology powered orbital telecommunications network I am unfamiliar with, the only similarity between them and technology is in the minds of the users, not the stuff itself.
- oreegumee 3y agoWhat we get out of these gadgets is addiction to peering into them for answers and filling an existential hole like astrology. Meanwhile tomorrow will go by just the same as yesterday. Technology isn’t altering physics, it’s operates within the boundaries set by experiment a century ago. It’s not curtailing human social problems that have existed since forever. Trade offs in society just like engineering; arguments it saves people from exploitation at home but airplanes and modern tracking enable large scale human trafficking. The yearly resource consumption for yearly phone updates is insane. If designer drugs that implanted experience made line go up that’s the technology we have. But what we do have makes line go up; why bother with other ideas? Social zeitgeist is to see consumer tech as allowing us to experience and achieve magical things as with astrology, yet daily life is still right there too. We’re addicted to juicing our own biochemistry with video games because drugs are bad mmmk
- a_e_k 3y agoSomething like that is the core idea behind Black Mirror S2E1, "Be Right Back" (2013). https://en.wikipedia.org/wiki/Be_Right_Back https://en.wikipedia.org/wiki/Be_Right_Back
- robotbikes 3y agoAlso something like this happens in the Greg Egan book Zendegi. It starts off with a anecdote about someone ripping their music files and the issues that arise from the copy (or something along those lines). But definitely an interesting read and one that tackles the possible issues that could arise in a different way than other novels.
- sir0010010 3y agoYou can already do this without LLMs: give someone else all of your loved one's writing samples and ask them to do their best to text you in a style as if it was the loved one.
- paxys 3y agoI'm not a professional in the field, but I really don't see how something like this could be therapeutic. Isn't the goal to help people let go and move on with their lives? To me this seems like the exact opposite.
- chrisco255 3y agoI think it would be ok from the right frame of mind. To think of the LLM as a digital memento and not as some kind of replacement for the deceased.
- orenlindsey 3y agoI wanted to do this myself but I don't have enough messages yet. Also, I wish the author shared some more technical information on how to do it.
- speps 3y agoThis repo, albeit "old" in regards to how much progress there's been in LLMs, has great simple tutorials right there eg. fine-tuning GPT2 with Shakespeare: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
- szmerdi 3y agoSimilar project with enough technical details: https://asmirnov.xyz/doppelganger https://asmirnov.xyz/doppelganger
- valray 3y agoAlexander Smirnov's article was discussed on HN: https://news.ycombinator.com/item?id=38434914 https://news.ycombinator.com/item?id=38434914
- iJohnDoe 3y agoIf you’re more an email person, then it might be a better resource? Need to have a process to extract information from Outlook .msg files? To: From: Body: Actually, I guess you could just do an export from Outlook to a CSV file and then process?
- ars 3y agoSetup a local IMAP server (for example Dovecot), have Outlook connect to it (I think you have to use Old-Outlook if on a Mac, not sure) and copy all your Outlook messages to it.
- Sohcahtoa82 3y agoI tried this myself using GPT-2J and nearly 15 years of IRC logs trimmed down to only contexts that include me. Each training item would be my message and the preceding 20 messages. Still ended up being an 80 MB log file of over 1M lines. The results were...not good at all. I ran an IRC bot that would watch the channel, and any time someone sent a message, it would send the last 20 messages to the trained model to see what it would predict the next message to be. It generated mostly garbage. Generated chat messages were on-topic, but non-sensical. It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. At some point, I'll probably try again with a far better model. Likely once I do my next system overhaul so I can do it locally. The only thing I'm not sure of is how to properly train and generate. GPT3 and its predecessors are very much simple text-completion AIs, whereas it seems all the models today are designed for a conversation back-and-forth between the AI and a user.
- jokethrowaway 3y ago> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)
- Sohcahtoa82 3y ago> I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.
- eightysixfour 3y ago
- zoklet-enjoyer 3y agoI remember having a similar idea near the end of high school. So 2006ish. I spent a lot of time online, on forums and instant messaging friends I knew in real life and just online. I saved all of my conversations because I thought it would be useful later on in life if I wanted to write a biography, read conversations like a diary, or analyze it with a computer program and create a chatbot. I saved text messages and emails because of this idea too. I don't have any of that data anymore. Lost most of it between 2010 and 2013.
- sjwhevvvvvsj 3y agoIt’s amazing how dogshit llama2 7b is in comparison with 13b (let alone 70b). The only 7b foundation model that is worthwhile is mistral imho.
- AndrewKemendo 3y agoI’m glad someone put the time it to show it can finally work. Would love to do this too, but have a gazillion other projects that are priority.
- 0xEF 3y agoI also considered it, but the existential dread of discovering myself to be an insufferable numpty pretty much quelled that fire.
- visarga 3y agoIt's probably sufficient to just interact as much as you can and save your data. It's going to get "uploaded" to a LLM eventually. I saved about 17 years worth of my messages. Looking at my messages I think I don't need to train the model to generate that text, but rather identify the personality traits and emotional stance of the author, and be aware of personal facts. So probably something like RLHF would work better than SFT. Assuming I have such a model I can further collect preference data to align it to my personality. Over time it might get good enough and I can officially bless it, if it ever gets to that level. Getting our social account providers to export all our data without restrictions might become a new right we have to have. I know some of them are locking up since 2023 (reddit and twitter).
- vunderba 3y agoI’m far from the first to think of this. Several people — perhaps inspired by creepy Black Mirror episodes — have tried to fine-tune an LLM on their SMS or WhatsApp history in an effort to create a simulation of themselves. It's a much older concept than Black Mirror. Ever since Markov chain IRC bots got popularized in the late 90s and early 2000s, people have been trying to train their virtual doppelgängers. I'm sure it won't be long before you can pay to have a post-Mortem ghost of yourself duplicated in a cemetery seance style MMO to perpetually live on and interact with other people who have passed on.
- HPMOR 3y agoThis actually seems like a really good startup idea.
- krainboltgreene 3y agoThis is definitely what silicon valley would consider a good startup idea.
- fnorder 3y agoI'm petty sure that Ray Kurzweil was working on this very thing when Google brought him on.
- mojomark 3y agoThe documentary with him saving his father's writings and such was one of the saddest things I've seen - that moment has stuck with me for years. I suppose the people who will be paying for such a service will be the loved ones left behind, not the one's dying. I wonder if the doppelgangers will start to talk to each other in the graveyard in the middle of the night?
- kevindamm 3y agoIf two digitized ghosts are talking but no air gets vibrated, do they make a sound?
- _spduchamp 3y agoI expect Google will one day try to sell us a similar service built from our Gmail accounts.
- monkeydust 3y agoWill first step be an upgrade to their autocomplete perhaps?
- recursive4 3y agoThis is similar to the origin story of Replika. The founding team were devastated by the death of their friend and wanted to immortalize him by using a language model.
- fractallyte 3y agoMy comment on something similar, 5 years ago (https://news.ycombinator.com/item?id=18546242 https://news.ycombinator.com/item?id=18546242): Better be careful with that... From "Bicycle Repairman", by Bruce Sterling (1996) (This is a spoiler, BTW!): "The mook speaks just like the Senator did, or the way the Senator used to speak, when he was in private and off the record. The way he spoke in his diaries. As far as we can tell, the mook was his diary.... It used to be his personal laptop computer. But he just kept transferring the files, and upgrading the software, and teaching it new tricks like voice recognition and speech-writing, and giving it power of attorney and such.... And then, one day the mook made a break for it. We think that the mook sincerely believes that it’s the Senator." Oh, and just note that this was way before Black Mirror.
- jacquesm 3y agoBut way after Gibson.
- justinlloyd 3y agoI did a fine tuning and embedding on a large LLM that is based on 50 years of daily journal entries, extensive daily notebooks usually measuring in the hundreds to thousands of words per day, and personal writings across a half-dozen different blogs and websites and various social media feeds. Social media posts and comments (including this one) are also put into my notebooks with a snippet of context about why I posted that. In my journal and daily notebooks I write out some pretty deep and dark thoughts too. It's about as close to a snapshot of my inner thoughts as you can imagine. And the LLM digested all of that. It took quite a while. It has provided some interesting insight into who I am as a person, and also how I see myself.
- Exuma 3y agoYou can’t leave us hanging like that … so what happened. What did you learn? Was it weird reading it? How do you see yourself?
- justinlloyd 3y agoI discovered that I can be a bit of a tease.
- romwell 3y agoGoddamit.
- ImHereToVote 3y ago+
- jakkos 3y agooutstanding
- cptcobalt 3y agowell done.
- legendofbrando 3y ago
- ilrwbwrkhv 3y agoWe need to learn to let go. Chatting with a deceased loved one is basically equivalent to the ressurection stone in Harry Potter. A faint reflection which will drive people to insanity. This is not healthy at all.
- justanotherjoe 3y agoThis is worrying for me. Imagine if someone, god forbid, is encouraged into suicide because he felt safer because he had 'left something behind'. Or like, wire the llm into his messenger to pretend as if he is still alive...
- jacquesm 3y agoHarry Potter was very late to that particular party. John Crowley, 'Snow' (1986) comes to mind as well as Gibson in 1984 with Neuromancer. And likely the concept is much older than that.
- prepend 3y agoI feel similarly. It’s emotionally dangerous because obviously we miss our loved ones. And hearing their voice or feeling a simulacrum of talking to them is probably comforting in some way. But it’s ultimately hollow. And can’t really answer anything new other than what’s sent into the training. Or, perhaps more dangerously, the simulation may hallucinate things that don’t exist and confuse people. (Eg, “I loved your sister more than you.” Or “I once had an affair in Paris”) I think what’s more tricky is after a breakup, training a bot in the ex’s chat history to simulate your ex and continuing on with “good morning/good night” chitchat. That probably really messes with people but will be chosen over the pain of rejection. I’m not sure how many messages are really needed, but it’s pretty easy to collect a few thousand texts over a short period of time.
- deleted 3y ago[deleted]
- Donz1 3y agoAmazing!!!! Can i write an article on it using your data?
- Roark66 3y agoI have a better idea... Have something like Google lens (remember these?) or another embedded camera record all your human interactions. Then we run audio to text on the voice and clip or other scene descriptions on the video(and ocr on text typed/read) and we fine tune a model on it. Imagine if you had 5+ years of such data (of someone that interacts with people a lot). Although "brain-links" are even more of a cool sci-fi idea. Imagine we can reliably "hook up" at synapse level. We first record everything for a year, then we plug a LLM into it so you can essentially "think together". Eventually you increase the size of the LLM as your wetware brain becomes unusable due to old age. Finally the biological brain dies, only the LLM remains (of comparable parameter size). Are you still alive, or is this a simulation? Of course I'm taking huge liberty in calling such AI a LLM. LLLm work on the basis of tokens and do not learn beyond their training phase, also one would need some input/output too. But imagine if this was figured out? Then you could live on as long as you can afford the power/maintenance bill for that datacenter that runs your mind :-)
- HPMOR 3y agoThis post motivated me to do the same thing on my imessage data. I used GPT-3.5 for fine-tuning (and haha GPT-4 to help me build it :D). I guess, I should run fine-tuning jobs to create instances of my correspondence partners? Since, for the first system prompt I tell the model to pretend to be me, but on other fine-tuning attempts I can reverse the input and output, and then tell the model to "pretend" to be my counterparty? I'm still running my first fine-tuning job but I'd definitely love any technical tips people want to drop!!
- Apfel 3y agoThis is a lot more risky than doing it on a self-hosted model surely? Do you really want to upload your entire message history to a third party (i.e. not Apple)?
- prepend 3y agoThis. I assume OpenAI is using all of our input to create digital twins of all users for purposes of marketing. And they’re probably better at it than I. So I might get a chatbot and OpenAI gets a marketing bot that fine tunes ads based on emotional state.
- TheOtherHobbes 3y agoWhy keep humans around at all? With only a little extra data and training effort it should be possible to create an entire internet of immortal bots living out shiny self-actualising Apple-perfect virtual lives, while the surviving hairless monkeys live out their short, pointless, ugly, imperfect existence slaving away in the lithium mines.
- hannofcart 3y ago> The LLM would generate conversations that seemed to be surreal parodies of my life. My work colleagues would endlessly reschedule meetings earlier and later; my partner and I would relentlessly debate our lunch options. Or as I call it, life.
- Borrible 3y agoAt least in Western magic and mythology, doppelgangers have been harbingers of personal doom for centuries - so beware of your evil twin. Just saying, so that no one is surprised when they start to doubt themselves...
- nonameiguess 3y agoPeople keep mentioning science fiction that this type of thing has been done in. Quick reminder since no one actually watched Caprica that this is how the Cylons started. Daughter of a dude who made a digital scanning technology to create VR avatars makes it better and her avatar becomes sentient. She dies as a suicide bomber and dude discovers the avatar and tries to bring his daughter back by building it a body. It becomes the first Cylon and humanity is extinguished from that particular solar system. The second Cylon turns out to be Admiral Adama's older sister who was also killed in the same suicide bombing. In any case, it's worth remembering that in reality people are more than what they write, speak, and otherwise ever put into language to communicate to or record for others. I imagine this particular tactic would not work well for me since I have sent single digit SMS messages in the past year and presumably nowhere near six figures as a lifetime number. You'd have to find some other source of writing, but I don't know what that would be. I'd guess at least 90% of the comments I even compose for Hacker News end up being deleted without ever posting them. Turning what I make public into a copy of me would give you a heavily censored, curated image of a fake person who is far more coherent and kind than the real me, and he would also seem to know everything since he's very good at web search. Really, I am quite surprised even reading old forum posts of mine from 20 years ago that I manage to track down, in part because of how much my personality and opinions have changed since then, but also at the impressive way I was able to digest and spew so much basically correct as far as I can tell information into coherent arguments even though I don't think I understood very much of it and remember even less. I'd expect most Hacker News posters to buy into the obvious truism that what you see on Instagram is not real people but that is equally true of text. Doing this kind of thing at-large risks creating simulacra of perfectly curated uber-humans that would make real people supremely inadequate. We're already going through a mental health crisis exacerbated by people feeling they can't live up to social media personas.