8 ms·
Markov chains are funnier than LLMs
- derefr 2y agoAn LLM trained on a given dataset should — at least in theory — "contain" (in a lossless-data-compression sense) a full superset of the knowledge of a Hidden Markov Model trained on the same dataset. I.e. that information is there, in the weights, in some form, and could in theory be used to reconstruct an equivalent HMM from the LLM. Why can't we get LLMs to do what HMMs do, then? Mostly, it comes down to the structure. Markov models are "funny" because they just have one level of abstraction: tokens. Markov "inference" is predicting the next token, given the last N tokens, and a model that knows weights for what tokens follow what N-tuples of previous tokens. And due to that limitation, the only rules that HMMs can learn, are low-level rules that don't require any additional abstraction: they can't optimize for syntactically-valid English, let alone semiotically logical statements; but they can make the text "feel" good in your head [i.e. the visual equivalent of song vocals having nice phototactics] — and so that's what training the model leads it to learn to do. And it turns out that that combination — text that "feels" good in its phrasing, but which is syntactically invalid — happens to read as "funny"! LLMs aren't under the same constraint. They can learn low-level and high-level rules. Which means that they usually do learn both low-level and high-level rules. The only thing stopping LLMs from using those low-level rules, AFAICT, is the architectures most LLMs are built on: the (multi-layer) Transformer architecture. Transformer LLMs are always a single-pass straight shot ("feed forward") through a bunch of discrete layers (individual neural networks), where at each step, the latent space (vocabulary) of the layer's inputs is getting paraphrased into a different latent space/vocabulary at the layer's outputs. This means that, once you get into the middle of a Transformer's layer sandwich, where all the rules about abstract concepts and semiotics reside, all the low-level stuff has been effectively paraphrased away. (Yes, LLMs can learn to "pass through" weights from previous layers, but there's almost always a training hyperparameter that punishes "wasteful" latent-space size at each layer — so models will only usually learn to pass through the most important things, e.g. proper names. And even then, quality on these "low-level" inferences are also the sorts of things that current test datasets on LLM ignore, leading to training frameworks feeling free to prune away these passthrough nodes as "useless.") This problem with LLMs could be fixed in one of two ways: 1. the "now it's stupid but at least it rhymes" approach Allow inference frameworks to simply bypass a configurable-per-inference-call number of "middle layers" of a feed-forward multi-layer network. I.e., if there are layers 1..N, then taking out layers K..(N-K) and then directly connecting layer K-1 to layer N-K+1. At its most extreme, with layer 1 connected to layer N, this could very well approximate the behavior of an HMM. Though not very well, as — given the relatively-meaningless tokenization approach most LLMs use (Byte Pair Encoding) — LLMs need at least a few transforms to get even to the point of having those tokens paraphrased into "words" to start to learn "interesting" rules. (AFAIK in most Transformer models layers 1 and N just contain rules for mapping between tokens and words.) Meanwhile, this would likely work a lot better with the "cut and graft" happening at a higher layer, but getting the "graft" to work would likely require re-training (since layers K-1 and N-K+1 don't share a vocabulary.) ...except if the LLM is an auto-encoder. Auto-encoder LLMs could just run an inference up their layerwise "abstraction hierarchy" to any arbitrary point, and then back down, without a problem! (I'd really love to see someone try this. It's an easy hack!) 2. the "it can write poetry while being smart" approach Figure out a way, architecturally, to force more lower-layer information from the early low-level to be passed through to the late low-level, despite the middle layers not having any reason to care about it. (I.e. do something to allow the LLM to predict a word Y at layer N-3 such that it rhymes with word X known at layer 3, while not otherwise degrading its capabilities.) Most simply, I think you could just wire up the model with a kind of LIFO-bridged layer chain — where every layer K is passing its output to the input of layer K+1; but, for any given layer K in the first half of the layers, it's also buffering its output so that it can become an additional input for its "matching" layer N-K. This means that all the layers in the "second half" of the model would receive longer inputs, these being the concatenation of the output of the previous layer, with the output of the matching "equal in abstraction depth" input layer. (Where this equal-in-abstraction-depth association between layers isn't inherently true [except in auto-encoder models], but could be made true in an arbitrary model by training said model with this architecture in place.) (Again, I'd really love to see someone try this... but it'd have to be done while training a ground-up base model, so you'd need to be Google or Facebook to test this.)
- ggm 2y agohttps://en.wikipedia.org/wiki/Mark_V._Shaney https://en.wikipedia.org/wiki/Mark_V._Shaney
- emmelaich 2y agoThe Policeman's Beard is Half-Constructed. https://archive.org/details/policemansbeardi0000unse https://archive.org/details/policemansbeardi0000unse
- superkuh 2y agoI've been running IRC markov chain bots for ~20 years. In the last couple years I've been running local LLM alongside them. While there are people that still prefer the markov chain bots the majority invoke the LLMs. But maybe that's because I picked the most hallucinate'y, least refusing, most funny models I can (like mistral-7B fine tunes) instead of something smart, locked-down, and low temperature like chatgpt's LLM as a service. If anything seeing the LLM and markov bots side by side has really reinforced how much of the markov bot "humor" is human perception imposed on chance outputs. The markov's "learning" ability is still far superior though.
- kdbg 2y agocurious what type prompting you do on the LLM? I run a markov chain bot in a Twitch chat, has some great moments. I tried using a LLM for awhile, would include recent chat in the prompting but never really got results that came across as terribly humorous, I could prompt engineer a bit to tell it some specifics about the types of jokes to build but the LLM just tended to always follow the same format.
- superkuh 2y agoI'm actually not following the model's fine-tuned/desired prompt at all. I am operating in purely pattern completion mode. The first text the LLM sees are alternating lines of input and response examples that look like what it will get getting from the IRC client front end written in the tone I want it to respond and giving some information about itself. Then I just tack the IRC chat history+input onto those example chat pre-prompt lines. Nothing but single lines and newlines with newline as a stop token. No instructions, nothing meta or system or the like. But that's also configurable by users. They can invoke any pre-prompt they want by a command passing a URL with a .txt file.
- ghewgill 2y ago> I've been running IRC markov chain bots for ~20 years. You are my hero. Mine have never lasted that long. One fun thing I did once was scrape user's livejournals and generate random text from them (https://hewgill.com/journal/entries/68-new-lj-toy.html https://hewgill.com/journal/entries/68-new-lj-toy.html).
- AdieuToLogic 2y ago> Asking an LLM for an “original thought” is almost oxymoronic, if not just moronic. It was built with the express purpose of not doing that. What a wonderful insight.
- make3 2y agoalso, not really true, right, even though it sounds intellectual and strong to say. these algorithms are trained to generalize as best as they can to unseen text, and most often don't ever see any data point twice, except for data that has accidentally not been filtered. it's totally possible that it gets reasoning abilities that generalize well.
- lossolo 2y agoGeneralize over their training data—they cannot generalize out of distribution. If they could, they would have already solved most human problems. So no, they do not generalize on unseen text. They will produce what is most statistically probable based on their training data. Things that are still unknown and statistically improbable based on our current knowledge are out of reach for LLMs based on transformers.
- make3 2y ago"generalize to its dataset" is a contradiction, especially as these models are trained in the one epoch regimen on datasets of the scale of all of the internet. if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm not sure what I can tell you
- lossolo 2y ago> "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm not sure what I can tell you If after reading papers below that show empirically that they can't reason, you will still think they can reason, then I don't know what I can tell you. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809 https://arxiv.org/abs/2309.01809
- complaintdept 2y agoMy favorite markov experiment was feeding it the works of Nietzsche from Project Gutenberg. Really brought out the late stage syphilis, but was still very distinctly Nietzsche.
- techjamie 2y agoI love messing with Markov chains, and before Reddit dunked their API, I would feed my Markov chain comments from NSFW subreddits and political subreddits at different sample sizes. I would spend ages just watching it whiplash about some political topic and sampling thirst comments.
- duskwuff 2y agoThe live feeds are all dead now, but you can still download archives of Reddit posts/comments from before they locked down the API: https://academictorrents.com/details/9c263fc85366c1ef8f5bb9da0203f4c8c8db75f4 https://academictorrents.com/details/9c263fc85366c1ef8f5bb9d...
- canjobear 2y ago“Talk to Transformer” GPT-2 was very funny, then GPT-3 was not funny at all. We blew through the uncanny valley in a couple of years.
- hatthew 2y agoI've posted a few "fake XYZ written by AI" on reddit over the years, and the model that got the best reactions was GPT-2. Markov chains aren't good enough to make something that is interesting for more than a sentence or two, and GPT-3 onwards is too clean and boring. GPT-2 is the perfect middle ground that can get grammar largely correct and maintain a cohesive idea, but doesn't yet know enough of the specifics of various topics to make things that make sense in the context of that topic.
- metadat 2y agoCan you share any of the best examples?
- PawgerZ 2y agohttps://www.reddit.com/r/SubSimulatorGPT2/top/?t=all https://www.reddit.com/r/SubSimulatorGPT2/top/?t=all This is all GPT2 generations trained on reddit data. https://www.reddit.com/r/SubSimulatorGPT2/comments/btfhks/what_is_rsubsimulatorgpt2/ https://www.reddit.com/r/SubSimulatorGPT2/comments/btfhks/wh... Here's the subreddit explained
- Sohcahtoa82 2y agoI used my 15+ years of IRC logs to fine-tune a GPT-2 model to try to imitate me. My plan was to deploy a bot in my regular IRC channel and see how long it took for people to notice it was a bot. Any time someone would send a message, it would send the last 10 messages to the LLM, and if the result started with <Sohcahtoa>, then I would send that message to the channel. Unfortunately, I quickly found GPT-2 isn't nearly good enough. It would generate slightly-coherent yet on-topic nonsense. Once I overhaul my system, I'll try fine-tuning a 7B model.
- loa_in_ 2y agoI'm pretty sure you could make it work if you expanded the context to hundred or so messages.
- bigyikes 2y agoTurn up the temperature? Markov chains have a cruder understanding of language. Turn up the temperature (the “randomness”) of an LLM and you can achieve a similarly crude approximation. Further, author uses ChatGPT-3.5. ChatGPT has been rlhf’d to sound as generic as possible, and 3.5 has a worse understanding of humor compared to 4. I don’t buy the thesis of this article.
- dartos 2y agoI wish there was a locked chatgpt version for researchers
- eru 2y agoSome of the open-weights models come close to what you might want?
- dartos 2y agoSort of. They fill the “locked” requirement, but open models aren’t SOTA like gpt or Claud.
- coder543 2y agoLlama3.1-405B benchmarks at a level that I would certainly consider comparable to other SOTA models, as long as multimodality isn’t needed. Regardless, OpenAI provides access to quite a few of their older models through the API, since the API lets you pass in a specific model version. I’m sure the older models won’t be available forever, but that is a much more stable target for researchers than just opening the ChatGPT website and typing in things.
- dartos 2y agoChatGPT specifically is more than just a chat interface on top of got-4. Their system prompt includes the current date and time among other information, making it very very hard to run reproducible experiments against it. But it’s the tool most people are using.
- dgacmu 2y agoSome friends of mine in college ran a markov chain generator on the "police reports" section of our college newspaper (1). The best 10% of the results, from a three-token generator, were some of the funniest machine-generated text I've ever seen -- it had an absurdity that modern LLMs seem to avoid due to making higher-level semantic sense. It probably also helped that there was a creep exposing himself in the library during this period, which made for some good base material. (1) The Daily Utah Chronicle; if memory serves, said friends also tried the markov chain generator on the personals section to good effect as well.
- gs17 2y ago> it had an absurdity that modern LLMs seem to avoid due to making higher-level semantic sense. That's it, LLMs are "trying" to be funny but aren't quite smart enough to actually be funny and their errors are just boring. Markov chains are accidentally hitting on absurdist bits because every sentence gets randomly brought in whatever the homograph equivalent to a malapropism is.
- trescenzi 2y agoReminds me of how the fake magic cards before chatgpt were always hilarious and sometimes playable, now they are usually playable and sometimes hilarious. Definitively a downgrade if you’d ask me.
- JoshTriplett 2y ago> always hilarious and sometimes playable I saw plenty of those back then, and as far as I could tell, examples were always cherry-picked from a larger set.
- kibibu 2y agoMaybe the right solution is using a Markov chain or gpt1 to generate ideas and an LLM to cherry pick.
- stochaztic 2y agoYes, I ran the RoboRosewater account, which is where most people saw the concept I believe. I had a few documents with thousands of cards of output, went through them for any notable ones, and formatted and posted them. Probably 5-10% of the output made my personal cut for notability.
- Applejinx 2y agoThen I salute you. Some of the funniest things I've EVER seen in my whole life was watching the LoadingReadyRun comedy troupe, doing their 'Desert Bus' fundraiser and punchy from lack of sleep, encounter RoboRosewater for the first time. It became a real tradition :)
- jsemrau 2y agoCan fully agree. Source me who ran a markov chain based Twitter autogen tool 8 years ago.
- donatj 2y agoHey, me too! I populated the markov chain with the text of my Twitter feed with a recency bias and generated tweets based on that so it stayed topical. The account sat unused after Twitter locked down their API, and at some point got hacked without me noticing. It had been taken over by a crypto scammer, and the account got banned. Trying to get it back was fruitless, Twitter/X's support is entirely useless.
- jsemrau 2y agoI got banned within 24 hours ^-^. Rightfully so, I assume. The Markov chain agent was supposed to send everyone who tweets "I am bored" an event recommendation based on their profile and location. I severely underestimated how many people tweet "I am bored"... Lessons learned, I guess.
- donatj 2y agoOh, I didn't @ or DM anybody. I just had a feed with tweet once an hour, and even then I manually approved the Tweets before they went out. My account made very clear it was a bot, it was in its name. Didn't ever get very many followers. I would say I never really got past the experimental phase.
- tracerbulletx 2y agoNo, they might be funnier than an LLM specifically aligned to not be that funny. Fine tune an LLM base model with jokes and align it by ranking how funny each reply is, instead of helpful questions and answers then we'll talk.
- renewiltord 2y agoThe bottomless pit copypasta remains unbeaten. https://www.reddit.com/r/greentext/comments/vc7hl0/the_bottomless_pit_supervisor/ https://www.reddit.com/r/greentext/comments/vc7hl0/the_botto...
- renewiltord 2y agoSome portion of this is that the prompt is good but I am entertained by so many of these. Humor is personal, it's true. But I found it quite funny. E.g. https://pastebin.com/84ByWUJL https://pastebin.com/84ByWUJL And another greentext for you: >Be me >Be a bottomless pit supervisor >Spend months yelling into the void >Echo never comes back >Start to think the pit is ignoring me >Decide to teach it a lesson >Dump truck full of Lego bricks into the pit >Ground starts shaking >Unholy scream erupts from the depths >mfw I'm actually a regular pit supervisor >First day on the job >Realize it's just the sewage treatment plant >Get fired for clogging entire city's plumbing
- dimatura 2y agoThe evolution of the AI weirdness blog (https://www.aiweirdness.com/ https://www.aiweirdness.com/) over the past few years somewhat supports this idea, except that the author got plenty of funny results with LLMs - but mostly the early ones, up to around GPT3 (and smaller variants of GPT3). For example, cereal names generated by the Ada version of GPT were much funnier than the Da Vinci ones: https://www.aiweirdness.com/new-breakfast-cereals-from-ai/ https://www.aiweirdness.com/new-breakfast-cereals-from-ai/
- jcalvinowens 2y agoUnfortunately the account no longer exists... but about a decade ago while I was in school, I created a Markov twitter bot trained on: 1. All of Linus Torvalds' mail to LKML for the prior year. 2. All of Jesus' direct quotes from the king james bible. It was absolutely hilarious. The two training sets had very little overlap, so it was necessary to add a heuristic that weighted options from each set more heavily the longer the chain had been "stuck" in the other set.
- throwawayk7h 2y agoThis sounds great. Did you save any examples?
- NobodyNada 2y ago"King James Programming" is along similar lines: https://www.tumblr.com/kingjamesprogramming https://www.tumblr.com/kingjamesprogramming
- throwawayk7h 2y agoYes, this is the first thing in the article.
- SonOfLilit 2y agoThis very similar bot still exists (and was quoted a few times in Unsong): https://www.tumblr.com/kingjamesprogramming https://www.tumblr.com/kingjamesprogramming
- hkgjjgjfjfjfjf 2y ago[dead]
- maxglute 2y agoI use to be unreasonably entertained tapping the suggest next word feature on google keyboard.
- meowface 2y agoMany years ago I trained a Markov chain chat bot on all of the conversation history in a group chat with me and my friends. It was regularly incredibly hilarious. I also would train it on subsets of them, like two of the friends but no one else, and name that version of the bot a combination of their names. Something about it was just so funny.
- tkgally 2y agoPrompt to Claude 3.5 Sonnet: Below is a recently published essay followed by some of the discussion about it on Hacker News. Write 10 short jokes on this theme, making sure that the jokes are funny in the way described by the essay's author. [essay and discussion omitted] Claude's response: I understand you'd like me to write 10 short jokes on the theme of Markov chains being funnier than large language models, in the style described by the essay author. I'll do my best to create some jokes that aim for "unserious surprise" and violate expected patterns in amusing ways: 1. Why did the Markov chain cross the road? To get to the other side of unpredictability. 2. An LLM and a Markov chain walk into a bar. The LLM orders a statistically probable drink. The Markov chain orders a lampshade made of cheese. 3. What do you call a Markov chain that tells dad jokes? A Mark-ov Twain. 4. How many LLMs does it take to screw in a lightbulb? One - but it'll spend 20 minutes explaining the optimal lightbulb-screwing process first. 5. Knock knock. Who's there? Markov. Markov who? Markov chain reaction of nonsensical hilarity. 6. An LLM, a Markov chain, and GPT-4 walk into a bar. The bartender says "We don't serve your kind here." GPT-4 leaves. The LLM stays to debate ethics. The Markov chain orders a bicycle made of spaghetti. 7. What's an LLM's favorite movie? Predictable and Furious 17: The Safest Driving Yet. 8. How does a Markov chain tell a story? "Once upon a time, there was a princess who lived in a castle made of bananas and ruled over a kingdom of sentient toasters." 9. What did the Markov chain say to the LLM? "Your mother was a abacus and your father smelled of silicon!" 10. Why don't Markov chains make good therapists? Because their advice is always "And then you should banana the feelings until they become a hat."
- anvuong 2y ago"An LLM, a Markov chain, and GPT-4 walk into a bar. The bartender says "We don't serve your kind here." GPT-4 leaves. The LLM stays to debate ethics. The Markov chain orders a bicycle made of spaghetti." This is actually gold.
- muglug 2y agoIt’s... not? Even for the low bar of a geek joke it makes no sense since GPT-4 is an LLM.
- tombert 2y agoI am in a private Discord server that has two bots in it. One is a bot that is a basic Markov chain trained on the entire chat history. The second is a proper LLM trained on some amount of tokens backward. Both will occasionally just randomly chime in during the chat. The markov chain bot is always considerably funnier.
- Sohcahtoa82 2y agoCurious what context window you used. My understanding is that a short window like only 1-2 words creates gibberish, but longer windows tends to repeat previous messages verbatim. And when deciding to chime in, was it just a simple chance (ie, 25%) after any other message? Or did it run on a timer?
- pants2 2y agoIf you want some empirical evidence: /r/subreddit simulator[1] is a Markov-based Reddit parody, and /r/SubSimulatorGPT2[2] is its LLM-powered cousin. The Markov version got way more upvotes and is simply funnier! 1. https://www.reddit.com/r/SubredditSimulator/top/?t=all https://www.reddit.com/r/SubredditSimulator/top/?t=all 2. https://www.reddit.com/r/SubSimulatorGPT2/top/?t=all https://www.reddit.com/r/SubSimulatorGPT2/top/?t=all
- mcmoor 2y agoNah, I think it's solely due to the former being much older and more well-known. And I always liked the latter much more anyway.
- Miraltar 2y agoI agree, Markov chain is fun when you don't know what it is because you try to make sense of it but once you realize it's just gibberish it's pointless. The other has that "unserious surprise" talked about in the original article.
- wcedmisten 2y agoHighly recommend this performance from the last Strange Loop on a similar topic! I had the pleasure of seeing it live. A great combination of live music and Markov chains. https://m.youtube.com/watch?v=M2o4f_2L0No https://m.youtube.com/watch?v=M2o4f_2L0No
- johntb86 2y agoLLMs can be funny. For example, look at Golden Gate Claude ( https://news.ycombinator.com/item?id=40459543 https://news.ycombinator.com/item?id=40459543 ). But they're not good at intentionally being funny, so we need to break them to get absurdist humor instead.
- cancerhacker 2y agoSurprised no mention of. “The Policeman’s Beard is half constructed”[1] - the first book “written” by a computer program. And I imagine the git-man-page-generator[2] is similarly built (and far more fun!) [1] https://archive.org/details/policemansbeardi0000unse https://archive.org/details/policemansbeardi0000unse [2] https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/
- cout 2y agoThey had their love, but they also had their typewriters.
- at_a_remove 2y agoI used to keep a text file on hand back when people had some irritating Markov chain bots on IRC. Fed enough of the lines, the bots veered off into some new valley of chatter, appearing quite mad. Used to get some "What did you DO to my bot?" indignant messages. I wonder if I still have that file somewhere.
- fishtoaster 2y agoI came to this same conclusion some years ago while working on a side project. Before anything LLM existed, I built a site[0] to generate fake "AWS Blog Posts." I trained a markov chain generator on all AWS announcement posts up to that point, copied the html + css of aws's standard blog posts, then glued them all together with some python + JS. It turned out, IMO, pretty funny! People familiar with AWS's blog posts would often get several sentences in before they realized they were looking at word-soup. When GPT was new, I looked into using that to "upgrade" it. I spent a weekend messing around with Minimaxir's gpt-2-simple generating blog posts based on AWS content. What I found was, ultimately, it was way less fun. The posts were far too realistic to be interesting. They read like totally-real blog posts that just happened to not be true. I realized then that the humor of those early markov generations was the ridiculousness. The point where, a few words or sentences in, you realized it was all nonsense. LLM's these days are too good for that - the text they generate is sometimes wrong, but rarely nonsense in a humorous way. Markov chain content was wrong in a "kid's say the darndest things" way, while modern LLMs are wrong in a "My uncle doesn't know basic geography" way. [0] https://totes-not-amazon.com/ https://totes-not-amazon.com/ - click any link to get a new one.
- fouronnes3 2y agoReminds me of git man pages generators. Extremely fun as well when you can't quite tell if it's real.
- majewsky 2y agoRef: https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/
- mdp2021 2y ago> modern LLMs are wrong in a "My uncle doesn't know basic geography" way Oh, certainly not. I have been using LLM chatbots extensively this week. If your uncle were not just confident while stating falsities, but uncapable of self-assessment, he would have been murdered.
- bitwize 2y agoWhenever someone attending the tech meetup I frequent gets a little too jazzed and excited about AI, I pull out my phone, open Emacs (I use termux, btw), bring in a corpus of text (an excerpt from A Christmas Carol is good), and say M-x dissociated-press. Then I say "What this is doing is predicting the next character based on statistical likelihood of the previous few characters based on thencorpus of text. And fundamentally, that's all ChatGPT does -- predicting the next symbol based on a statistical model. ChatGPT has a much more sophisticated statistical model than this simple Markov chain and a vastly larger corpus, but really it's just doing the same thing." And we have a giggle about the nonsense DP makes of Dickens, but then I say that ChatGPT emits nonsense too, but it's far more insidious nonsense because it is much more plausible sounding.
- ineptech 2y agoRelatedly, the author Robert Anton Wilson tried to manually implement a Markov-chain-bot in the 1960s by cutting up passages from various books, selecting them from a hat, and looking for overlapping words and phrase fragments. The best results were included in the Illuminatus! trilogy, some attributed to a character named Markoff Chaney.
- 6510 2y ago> I’ve heard people claim that comedy cannot be generated by an algorithm. If it wasn’t obvious, I disagree with this. I think comedy can be analyzed and measured. I think, given a large enough grant, we could probably generate comedy on demand. I think it’s doable - not that we should do it. I use to think it came naturally, then someone had a book case full of books about humor. (wtf?) Apparently they have it down to a science. I learn the difference between someone funny and a professional comedian is that the later finds additional punch lines for a joke. It then described a step by step process going from a silly remark to a birthday joke comparing various modular developments into a kind of dependency hell complete with race conditions until the state object is carefully defined and the plot has the punchlines all sorted from the barely funny to the truly hilarious. It was more engineering than CS. The funniest seeBorg message was 10 minutes after a heated discussion that resulted in tanktop, a moderator, getting banned from a project. The bot wrote: Tanktop is Hitler! At that point it took 2 days for the humans to figure out what the next word was suppose to be.
- 29athrowaway 2y agoIt's all connected, bro. Language models are connected to compression, to n-gram models, to Markov chains, to everything.
- magicalhippo 2y agoI've also enjoyed the Postmodernism essay generator[1], which uses[2] a recursive transition network[3] to generate the text. [1]: https://www.elsewhere.org/journal/pomo/ https://www.elsewhere.org/journal/pomo/ (refresh for new, random essay) [2]: https://www.elsewhere.org/journal/wp-content/uploads/2005/11/tr-cs96-264.pdf https://www.elsewhere.org/journal/wp-content/uploads/2005/11... [3]: https://en.wikipedia.org/wiki/Recursive_transition_network https://en.wikipedia.org/wiki/Recursive_transition_network
- atoav 2y agoThe cool thing about market chains is that you can tune the probability with which it will jump between "genres" of it's training-content by making the analyzed N-Grams shorter or longer. E.g. it starts out as a news headline and ends with a bible verse.
- _emacsomancer_ 2y agoFunnily enough, I was idly playing with a last-updated-12-years-ago Markov chain text generator[0] in Emacs, and had quickly a number of amusing Markov chain outputs from mash-ups of the Elisp manual and Alice in Wonderland, including: "13.7 Anonymous Functions Although functions are usually defined with the built-in defmacro macro, but any list that begins with an M--' `Why with an M?' said Alice. `Why not?' said the March Hare." [0]: https://github.com/skeeto/markov-text https://github.com/skeeto/markov-text
- FeepingCreature 2y agoIsn't that the sort of joke that you'd get from GPT, not a Markov chain? Unless it's quoting verbatim, a Markov chain should have no idea that Alice is supposed to refer back to the letter from the previous sentence.
- StilesCrisis 2y agoThis phrase turns out to be a verbatim quote from the story.
- drivers99 2y agoSo it is: > “They were learning to draw,” the Dormouse went on, yawning and rubbing its eyes, for it was getting very sleepy; “and they drew all manner of things—everything that begins with an M—” > “Why with an M?” said Alice. > “Why not?” said the March Hare. > Alice was silent.
- tonetegeatinst 2y agoI just read a little bit into the post and though hey, let's call Markov chain's something better than a tiny llm. I present to hacker news the MCLM, the Markov chain language model.
- larodi 2y agoMarkov chains are funnier for one very basic reason: you can teach them to 5th or 9th graders. LLMS you can not, or at least it will take insane amount of allegory to do so. Markov chains are very tightly related regex, and one may be surprised that there is a probabilistic regex. Also to the graphical structure of Markov chains is a lot like a FSM, and FSM perhaps can be explained to very small children :D
- notjulianjaynes 2y agoIn college (around 2012ish) I made some poems by transcribing a recording of myself mumbling gibberish into Dragon Naturally Speaking, which partially relied on Markov chains for its ASR. It would output some string of text, and then I'd delete all the words it output from its dictionary; repeat. It worked very well for creating a list of (slant) rhymes, I have not found a better way to do this. As a literary tool, I've found LLMs to be seriously lacking in the ability to fuck around like this.
- raister 2y agoFor anyone interested, I've wrote a while ago this book: "Markov Chains for programmers". Link: https://czekster.github.io/markov/ https://czekster.github.io/markov/
- treprinum 2y agoGenerative AI often uses Markov chains as its theoretical basis and is modeled in the language of conditional probabilities albeit not stuck to a single ancestor but to a chain of ancestors. One can often view LLMs as scaled out Markov chains (more like "Erlang" chains allowing deeper memory) that encode the probability of the next token based on the past of the size of the context window and then doing some magic when picking next tokens.
- Jensson 2y agoEven old Markov chains has a context window of a few words, single word context Markov chains produce horrible results. People who don't think LLMs are Markov chains are just ignorant, not realizing that Markov chain isn't an algorithm, you can compute the probability in any manner and it is still a Markov chain.
- throwthrowuknow 2y agoIt’s not an n-gram, go learn about transformers
- Jensson 2y agoA context window is an n-gram.
- throwthrowuknow 2y agoIt’s not used as an n-gram
- ku1ik 2y agoThis is something people miss when talking about LLMs: „It’s the most average thing you could have possibly produced given the context.” They can’t be creative by design. They’re useful when you want to reproduce, but not when you want to create something completely new (that you can maybe do by getting a bunch of average outputs from an LLM and getting inspired yourself). When GPT-4 came out I was playing with it, and I often tried to get some unique, creative output from it, but very soon I learned it was futile. It was back when it all still felt magical, and I guess many of us tried various things with it.
- Kim_Bruning 2y agoIf you tell GPT-4 to get snarky, you can sometimes end up laughing (and it seems a tad more intelligent and socially perceptive besides!) . But Claude-3.5 can definitely get me laughing, at least. Now imagine telling Claude-3.5 to try being snarky while sorting out software issues at a customer's office. There should be a warning label!
- benob 2y agoLLMs are high-order Markov chains with a factored estimator. N-gram LMs are also Markov chains with a different estimator.
- yarg 2y agoUnserious surprise summarised it nicely for me; It does not mean that Markov chains are better - something trained to make predictions should ideally not fall too far away from our own internal prediction engines (which have been honed across aeons). It's that it starts to come close that's the problem (or cause); it's the uncanny valley for text.
- paweladamczuk 2y agoAm I the only one feeling uneasy when the Bible is used for funny experiments like this? Would you use an image of Christ on the cross to test an AI image modification model?
- nly 2y agoI'm sure most people here would given that like 50% of the world don't believe in an Abrahamic religion, and a % of those that do have a sense of humour.
- paweladamczuk 2y agoI wouldn't. And not because I believe it or lack a sense of humour.
- FeepingCreature 2y agoAs probably the oldest literary work that is also widely known in English speaking territory, the Bible has a highly distinct style that helps its samples maintain coherence in a sentence and also be easily recognizable.
- Kerb_ 2y agoI've used multiples images of Christ to play with AI image models. Remaking the old "Jesus on toast" image was fun. Facebook is full of boomers sharing the Jesus face AI generated into scenes. I feel uneasy when religion is used as an excuse to discriminate against or murder innocent people, not when symbolism or texts are used for creative purposes.
- fsndz 2y agothe original stochastic parrot
- dghf 2y agoI remember the KJV/SICP mash-up. My favourite, both for funniness and emotional truth: > And Satan stood against them in the global environment.
- wegfawefgawefg 2y agoThis is like saying its fundamentally impossible to make a fast car, but your only experience with cars is on roads with speed limits. People keep forgetting that the "safety", rlhf, and corpo political correctness post training is intentionally used to remove the funny from all the large models. The truth is we don't know if llms are funny or not. GPT2 was funny. GPT3 was funny before it was clockwork oranged. Everything after that is gimped. Even the open source models these days get rlhf'd in some way.
- PaulRobinson 2y agoThat's simply not true. GPT-3 was terrible at jokes. Many LLMs are because it's hard to model puns in a stochastic parrot. As another thread points out here, some LLMs produce marginally better jokes. As to your little range on "Political correctness" - that phrase just means "being polite". It does not mean "remove humor". It means "remove responses offensive to marginalized groups in society". Good humor "punches up", not down, so would not have any impact on good humor.
- Jensson 2y agoPolitical correctness doesn't mean polite, there were/are a lot of polite racists.
- PaulRobinson 2y agoIt is impossible to be polite to a group of people while also insisting that they are lesser humans and deserve a lesser place in society.
- jrflowers 2y agoThis is great logic. If it is impossible for a person to be both racist and polite then the solution to racism is everybody remembering to say please, thank you and pardon me when applicable. All social ills can be treated through decorum, hence why you never hear about bigotry amongst those that have been raised to adhere to strict social graces, such as the British aristocracy for example.
- throwthrowuknow 2y agoStopped reading at the mention of ChatGPT 3.5 for two reasons 1) if you’re not using the better model then you either don’t know enough for me to care about your opinion or you’re deliberately deceiving your audience in which case I’m not going to allow your meme pollution into my mind. 2) you are using the AI equivalent of a call centre support agent, they aren’t allowed to say anything funny. Most of their RLHF training has been specifically about NOT saying the funny things that will instantly go viral and cause a lot of media attention that will annoy or scare away investors.
- hrgafiI 2y agoBro, you have to use ChatGPT-4, or was it ChatGPT-4o, or was it ChatGPT-4p? Or was it Claude-3.5.1.2.3? I see a new phenomenon of AI "power users" emerging.
- throwthrowuknow 2y agoRight, who cares what version of a dependency you’re using? Those numbers don’t mean anything.
- Alifatisk 2y agoWasn't Microsoft Tay built using Markov chain?
- nwoli 2y agoBase model LLMs are funnier than instruction finetuned
- monokai_nl 2y ago23 years ago I created a web app (That Can Be My Next Tweet) that could generate your future tweet by feeding all your past tweets in a Markov chain. When the Twitter API access was shut down, I took it offline. At the time it went super viral for a while, maxing out to 2 tweets per second. It got some press and just now I went back to a Ted Talk of Adam Ostrow (Mashable), briefly showcasing this web app. He stated: you can imagine what something like this can look like 5, 10 or 20 years from now, and hinted at hyper-personalized communication AIs. By no means was my web app any foundation of the LLMs today, but it's interesting nonetheless how relatively simple techniques can trigger ideas of how future scenarios could look like.
- macNchz 2y agoSomething about the not-quite-rightness of Markov output has always really amused me, like I can’t help but giggle at it. My all-time favorite in this vein was @erowidrecruiter on Twitter, which generated posts with Markov chains from a corpus of tech recruiter emails and drug experience reports from erowid.org. Still up but no longer posting: https://x.com/erowidrecruiter?lang=en https://x.com/erowidrecruiter?lang=en
- Miraltar 2y agoI wouldn't say that LLMs aren't fun, it's more that jokes out of context aren't fun. If you ask somebody to just tell you a joke, there's a very low chance that you'll actually laugh, especially if it's in text format. But I've had some serious laugh looking at some of dougdoug videos featuring LLMs
- killerstorm 2y agoA small language model can be quite like Markov chain, e.g. the one from Karpathy's 2015 post was pretty nonsensical: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ So you can just scale down if it still makes sense. Also you get a lot more from the base model. GPT-3 was versatile as it could continue any context. Modern LLMs are try-hards. If you want to generate humor with LLM really worth going for base model with multiple examples in the prompt.
- vitus 2y agoApparently the ChatGPT output is ripped off of a 20-year old emacs joke. > In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. > – OliverScholz on news:alt.religion.emacs, 2003-03-28 https://www.emacswiki.org/emacs/TheBeginning https://www.emacswiki.org/emacs/TheBeginning (edited for brevity)
- rc_mob 2y agoIsn't LlLM just a fancier markov chain anyway?
- jablongo 2y agoThis person should’ve tried adjusting the temperature to their LLM before making this pronouncement
- fidotron 2y agoWe seem to be encountering more phenomena in tech where the increasing computing demands to improve the output exceeds any exponential growth in computing capacity. Graphics was the poster child for this, for years, and now the whole ML and AI space has discovered big machines make huge differences. Markov chains are almost trivial to implement and run on small devices. A slightly extreme example is a rock, paper, scissors game I did that worked this way: https://luduxia.com/showdown/ https://luduxia.com/showdown/ The actual browser side markov chain implementation of that took something like 2-3 hours.
- shagie 2y agoI still find https://www.infiniteconversation.com https://www.infiniteconversation.com to be one of the odd / funny / "uh... what?!" type things. It's not intended to be funny in the "ha ha" way but sometimes tickles my funny bone with the absurdist philosophy - talking about everything and nothing at the same time. From the FAQ it is a tuned LLM. > Mostly using open source tools available to anyone. The generation of the script itself is done using a popular language model that was fine-tuned on interviews and content authored by each of the two speakers.
- hellointernet 2y agoI used to have a markov-chain twitter bot that was trained on a few of my friends twitter posts. It was funny because that included hastags, tagged users, and images. There were some funny results and this was long before LLMs.
- drusepth 2y agoI posted "How to Speed Up Your Computer Using Google Drive as Extra RAM" [0] ~10 years ago and have had hundreds of people over the years ask if I used Markov chains to generate the funny bits (I didn't). I've had 0 people ask if I used LLMs on any of my comedy pieces. Seems QED to me. [0] http://drusepth.net/series/how-to-speed-up-your-computer-using-google-drive-as-extra-ram/ http://drusepth.net/series/how-to-speed-up-your-computer-usi...
- junon 2y agoA discord server I'm in is basically a shitpost/absurdist humor discord full of smart people, and we have at least 15 bots written by members that randomly engage with Markov chains built on the corpus of member chat history. They spout out some of the most unhinged, hilarious stuff. Always a good time. An LLM would struggle, I'd think, given that the humor usually stems from disjoint phrases that somehow take on new meaning. They're rarely coherent but often hilarious.
- ineedaj0b 2y ago>The best jokes involve a pleasant and significant “snap”. I use “snap” instead of “punchline” to avoid the semantic baggage. i like snap better too. it's more close to 'snapping the neck of the weak and feeble' which i think really embodies the spirit of joke tellers.
- hermitcrab 2y agoI wrote a little program to generate text from different text sources using Markov chains: https://successfulsoftware.net/2019/04/02/bloviate/ https://successfulsoftware.net/2019/04/02/bloviate/ Some of the output was moderately amusing. And text generated from Trump speeches by a Markov chain sounded very similar to a genuine Trump speech.
- anthk 2y agoNo references to Hailo/Megahal? https://homepage.kranzky.com/megahal/Classic.html https://homepage.kranzky.com/megahal/Classic.html
- lxe 2y agoThis article fails to mention sampling strategies that make LLM token generation incredibly unpredictable.
- npteljes 2y ago>These 2 examples are somewhat cherry-picked That's for sure. I have seen many Markov chain implementations, and if you could generate 1 funny thing for every 10 tries, then that was a good day. Both Markov chains and LLMs have a distinct style, which gets old over time. Markov much faster for me. So, in my experience LLMs win, by far. I do agree with the author that the LLM style can get really boring. I experienced the same myself. But the Markov results, while much less restrained, are so much more nonsense too. Often questioning its overall usefulness. Which, I think, the world also agrees on: while Markov chain implementations were fun toys at best, which worked sometimes to a kind of funny degree, LLMs are everywhere.
- ideashower 2y agoIs this was /r/subredditsimulator was built off of back in the day on Reddit?