5 ms·
I fed 24 years of my blog posts to a Markov model
- swyx 11mo agonow i wonder if you can compare vs feeding into a GPT style transformer of a similar Order of Magnitude in param count..
- 0_____0 11mo agoI thought for a moment your comment was the output of a Markov chain trained on HN
- bitwize 11mo agoNo mention of Rust or gut bacteria. Definitely not.
- fragmede 11mo agoThat's the question today. Turns out transformers really are a leap forwards in terms of AI, whereas Markov chains, scaled up to today's level of resources and capacity, will still output gibberish.
- atum47 11mo agoI usually have this technical hypothetical discussions with ChatGpt, I can share if you like, me asking him this: aren't LLMs just huge Markov Chains?! And now I see your project... Funny
- pavel_lishin 11mo ago> I can share if you like Respectfully, absolutely nobody wants to read a copy-and-paste of a chat session with ChatGPT.
- atum47 11mo agoWhen you say nobody you mean you, right? You can't possible be answering for every single person in the world. I was having a discussion about similarities between Markov Chains and LLMs and short after I found this topic on HN, when I wrote "I can share if you like" was as a proof about the coincidence.
- deleted 11mo ago[deleted]
- matusp 11mo agoLLMs are indeed Markov chains. The breakthrough is that we are able to efficiently compute well performing probabilities for many states using ML.
- famouswaffles 11mo agoLLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.
- sophrosyne42 11mo agoWell LLMs aren't human brains, unless you contort the definition of matrix algebra so much you could even include them.
- ben_w 11mo agoQM and GR can be written as matrix algebra, atoms and electrons are QM, chemistry is atoms and electrons, biology is chemistry, brains are biology. An LLM could be implemented with a Markov chain, but the naïve matrix is ((vocab size)^(context length))^2, which is far too big to fit in this universe. Like, the Bekenstein bound means writing the transition matrix for an LLM with just 4k context (and 50k vocabulary) at just one bit resolution, the first row (out of a bit more than 10^18795 rows) ends up with a black hole >10^9800 times larger than the observable universe.
- sophrosyne42 11mo agoYes, sure enough, but brains are not ideas, and there is no empirical or theoretical model for ideas in terms of brain states. The idea of unified science all stemming from a single ultimate cause is beautiful, but it is not how science works in practice, nor is it supported by scientific theories today. Case in point: QM models do not explain the behavior of larger things, and there is no model which gives a method to transform from quantum to massive states. The case for brain states and ideas is similar to QM and massive objects. While certain metaphysical presuppositions might hold that everything must be physical and describable by models for physical things, science, which should eschew metaphysical assumptions, has not shown that to be the case.
- roarcher 11mo ago...are you under the impression that you have an exclusive relationship with "him"? Everyone else has access to ChatGPT too.
- atum47 11mo agoYes. Yes I was. Thank you for the wake up call. I was under the impression that he was talking only to me.
- deleted 11mo ago[deleted]
- atum47 11mo agoDon't know what happened. I stumbled onto a funny coincidence - me talking to a LLM about its similarities with MC - decided to share on a post about using MC to generate text. Got some nasty comments and a lot of down votes. Even though my comment sparked a pretty interesting discussion. Hate to be that guy, but I remember this place being nicer.
- roarcher 11mo agoEver since LLMS became popular, there's been an epidemic of people pasting ChatGPT output onto forums (or in your case, offering to). These posts are always received similarly to yours, so I'm skeptical that you're genuinely surprised by the reaction. Everyone has access to ChatGPT. If we wanted its "opinion" we could ask it ourselves. Your offer is akin to "Hey everyone, want me to Google this and paste the results page here?". You would never offer to do that. Ask yourself why. These posts are low-effort and add nothing to the conversation, yet the people who write them seem to expect everyone to be impressed by their contribution. If you can't understand why people find this irritating, I'm not sure what to tell you.
- pavel_lishin 11mo agoNobody was being nasty. roarcher explained why people reacted the way they did.
- deleted 11mo ago[deleted]
- lacunary 11mo agoI recall a Markov chain bot on IRC in the mid 2000s. I didn't see anything better until gpt came along!
- nurettin 11mo agoYes, I made one using bitlbee back in the 2000s, good times!
- pavel_lishin 11mo agoI made one for Hipchat at a company. I can't remember if it could emulate specific users, or just channels, but both were definitely on my roadmap at the time.
- lloydatkinson 11mo agoI'm hoping someone can find it so I can bookmark it but I once read a story about a company that let multiple markov chain bots loose in a Slack channel. A few days later production went down because one of them ran a Slack command that deployed or destroyed their infrastructure.
- frumiousirc 11mo agoPerhaps you are thinking of megahal https://homepage.kranzky.com/megahal/Index.html https://homepage.kranzky.com/megahal/Index.html or if a bit later in the millennium, cobe https://teichman.org/blog/ https://teichman.org/blog/
- lacunary 11mo agoah, probably so, looks like there were eggdrop scripts for megahal, thanks!
- vunderba 11mo agoI did something similar many years ago. I fed about half a million words (two decades of mostly fantasy and science fiction writing) into a Markov model that could generate text using a “gram slider” ranging from 2-grams to 5-grams. I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random page in an old 1950s Funk & Wagnalls dictionary and using whatever I found there as a writing seed.
- bitwize 11mo agoTerry Davis, pbuh, did something very similar!
- davely 11mo agoI gave a talk in 2015 that did the same thing with my tweet history (about 20K at the time) and how I used it as source material for a Twitter bot that could reply to users. [1] It was pretty fun! [1] https://youtu.be/rMmXdiUGsr4 https://youtu.be/rMmXdiUGsr4
- idiotsecant 11mo agoDid it work?
- vunderba 11mo agoSo that's the key difference. A lot of people train these Markov models with the expectation that they're going to be able to use the generated output in isolation. The problem with that is either your n-gram level is too low in which case it can't maintain any kind of cohesion, or your n-gram level is too high and it's basically just spitting out your existing corpus verbatim. For me, I was more interested in something that could potentially combine two or three highly disparate concepts found in my previous works into a single outputted sentence - and then I would ideate upon it. So I haven't opened the program in a long time so I just spun it up and generated a few outputs: A giant baby is navel corked which if removed causes a vacuum. I'm not sure what the original pieces of text were based on that particular sentence but it starts making me think about a kind of strange void harkonnen with heart plugs that lead to weird negatively pressurized areas. That's the idea behind the dream well.
- anthk 11mo agoMegahal/Hailo (cpanm -n hailo for Perl users) can still be fun too. Usage: hailo -t corpus.txt -b brain.brn Where "corpus.txt" should be a file with one sentence per line. Easy to do under sed/awk/perl. hailo -b brain.brn This spawns the chatbot with your trained brain. By default Hailo chooses the easy engine. If you want something more "realistic", pick the advanced one mentioned at 'perldoc hailo' with the -e flag.
- hilti 11mo agoFirst of all: Thank you for giving. Giving 24 years of your experience, thoughts and life time to us. This is special in these times of wondering, baiting and consuming only.
- hexnuts 11mo agoI just realized, one of the things that people might start doing is making a gamma model of their personality. I won't even approach who they were as a person, but it will give their Descendants (or bored researchers) a 60% approximation of who they were and their views. (60% is pulled from nowhere to justify my gamma designation, since there isn't a good scale for personality mirror quality for LLMs as far as I'm aware.)
- jacquesm 11mo ago"Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart;" "Do me a favor, boy. This scam of yours, when it's over, you erase this god-damned thing."
- sebastianmestre 11mo agoCool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py < smach. I worgene arkov ment by compt the fecompultiny of 5, ithe dons Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferessiblementate > how examples, on 13 Debian is the more M-x: Execute testeration LLMs usually do some sort of tokenization step prior to learning parameters. So I decided to try out order-1 Markov models over text tokenized with byte pair encoding (BPE). Trained on TFA I got this: > I Fed by the used few 200,000 words. All comments were executabove. This value large portive comment then onstring takended to enciece of base for the see marked fewer words in the... Then I bumped up the order to 2 > I Fed 24 Years of My Blog Posts to a Markov Model > By Susam Pal on 13 Dec 2025 > > Yesterday I shared a little program calle... It just reproduced the entire article verbatim. This makes sense as BPE removes any pair of repeated tokens, making order-2 Markov transitions fully deterministic. I've heard that in NLP applications, it's very common to run BPE only up to a certain number of different tokens, so I tried that out next. Before limiting, BPE was generating 894 tokens. Even adding a slight limit (800) stops it from being deterministic. > I Fed 24 years of My Blog Postly coherent. We need to be careful about not increasing the order too much. In fact, if we increase the order of the model to 5, the generated text becomes very dry and factual It's hard to judge how coherent the text is vs the author's trigram approach because the text I'm using to initialize my model has incoherent phrases in it anyways. Anyways, Markov models are a lot of fun!
- countWSS 11mo agothe trick to prevent 'dry' output that quotes verbatim is to make the 5 words limit flexible: if there is only one path, reduce it to 4.
- Tallain 11mo agoI have a pet tool I use for conlang work for writing/worldbuilding that is built on Markov chains and I am smacking my forehead right now at how obvious this seems in hindsight. This is great advice, thank you.
- Aperocky 11mo agoHere's a quick custom markov page you can have fun with, (all in client) https://aperocky.com/markov/ https://aperocky.com/markov/ npm package of the markov model if you just want to play with it on localhost/somewhere else: https://github.com/Aperocky/weighted-markov-generator https://github.com/Aperocky/weighted-markov-generator
- anthk 11mo agoHailo from CPAN (Perl) it's much lighter than any NPM solution.
- ikhatri 11mo agoWhen I was in college my friends and I did something similar with all of Donald Trump’s tweets as a funny hackathon project for PennApps. The site isn’t up anymore (RIP free heroku hosting) but the code is still up on GitHub: https://github.com/ikhatri/trumpitter https://github.com/ikhatri/trumpitter
- monoidl 11mo agoI think this is more correctly described as a trigram model than a Markov model, if it would naturally expand to 4-grams when they were available, etc, the text would look more coherent Iirc there was some research on "infini-gram", that is a very large ngram model, that allegedly got performance close to LLMs in some domains a couple years back
- Legend2440 11mo agoGoogle made some very large ngram models around twenty years ago. This being before the era of ultra-high-speed internet, it was distributed as a set of 6 DVDs. It achieved state-of-the-art performance at tasks like spelling correction at the time. However, unlike an LLM, it can't generalize at all; if an n-gram isn't in the training corpus it has no idea how to handle it. https://research.google/blog/all-our-n-gram-are-belong-to-you/ https://research.google/blog/all-our-n-gram-are-belong-to-yo...
- tqian 11mo agoI have this DVD set in my basement. Technically, there are still methods for estimating the probability of unseen ngrams. Backoff (interpolating with lower grams) is an option. You can also impose prior distributions like a Bayesian so that you can make "rational" guesses. Ngrams are surprisingly powerful for how little computation they require. They can be trained in seconds even with tons of data.
- manthangupta109 11mo agoDamn interesting!
- OuterVale 11mo agoReally fascinating how you can get such intriguing output from such a simple system. Prompted me to give it a whirl with the content on my own site. https://vale.rocks/micros/20251214-0503 https://vale.rocks/micros/20251214-0503
- kazinator 11mo agoWhen the order argument is cranked up to 4, it looks to the average LLMvangelist like it is thinking.
- andai 11mo agoIn 2020, a friend and I did this with our mutual WhatsApp chat history. Except instead we fine-tuned GPT-2 instead. (As was the fashion at the time!) We used this one, I think https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple I think it took 2-3 hours on my friend's Nvidia something. The result was absolutely hilarious. It was halfway between a markov chain and what you'd expect from a very small LLM these days. Completely absurd nonsense, yet eerily coherent. Also, it picked up enough of our personality and speech patterns to shine a very low resolution mirror on our souls... ### Andy: So here's how you get a girlfriend: 1. Start making silly faces 2. Hold out your hand for guys to swipe 3. Walk past them 4. Ask them if they can take their shirt off 5. Get them to take their shirt off 6. Keep walking until they drop their shirt Andy: Can I state explicitly this is the optimal strategy
- Tepix 11mo agoThat‘s funny! Now imagine you‘re using Signal on iOS instead of WhatsApp. You cannot do this with your chat history because Signal won‘t let you access your own data outside of their app.
- GarnetFloride 11mo agoThe one author that I think we have a good chance of recreating would be Barbara Cartwright. She wrote 700+ romance novels all pretty much the same. It should be possible to generate another of her novels given that large a corpus.
- LanceH 11mo agoI'm not sure how we'd know. My wife sometimes buys and rereads a novel she's already finished.
- user_7832 11mo agoI can’t believe no one’s mentioned the Harry Potter fanfic written by a Markov Chain. If you’re familiar with HP, I highly recommend reading Harry Potter and the Portrait of What Looked Like a Large Pile of Ash. Here’s a link: https://botnik.org/content/harry-potter.html https://botnik.org/content/harry-potter.html
- mattacular 11mo agoGenuine question: Why would anyone want to read that? I glanced at the first sentence and decided not to go any further. It is hollow text. It has no properties of what I'd want to get out of even the worst book produced by human minds. Even more sophisticated models have a ceiling of pablum.
- grahamnorton39 11mo agoWhile hollow, it is also bad (and absurd) enough to be quite entertaining. It’s from an era where this wasn’t far off the state of the art for coming up with machine-generated text—context that makes it quite a bit funnier than if it were generated by an LLM today. That said, it’s obviously not to everyone’s tastes!
- user_7832 11mo agoIt's purely meant to be an absurdist read. It obviously makes no sense, yet is close enough to actual language patterns for (some, at least) people to find it hilarious. I had tears in my eyes from laughing way too hard when I first read it.
- cluckindan 11mo agoThat reads as coherently as a pile of Zen koans.
- mamma5211 11mo ago[dead]
- Peteragain 11mo agoSo, are current LLMs better because artificial neural networks are better predictors than Markov models, or because of the scale of the training data? Just putting it out there..
- microtonal 11mo agoMarkov models usually only predict the next token given the two preceding tokens (trigram model) because the data gets so exceptionally sparse beyond that, that it becomes impossible to make probability estimations (despite back-off, smoothing, etc.). I recommend you to read Bengio et al.’s 2003 paper which describes this issue in more detail and introduces distributional representations (embeddings) in an RNN to avoid this sparsity. While we are using transformers and sentence pieces now, this paper aptly describes the motivation underpinning modern models. https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
- Peteragain 11mo agoThanks for the reference and I stand corrected. And yes I had looked at it a long time ago and will give it another read. But I think it is saying that RNNs are a means of approximating a statistical property of a collection of text. That property is what we today think of as "completion"? That is, glorified auto complete, and not "distributed representations" of the world. Would you agree?
- microtonal 11mo agodistributed representations Distributional representations, not distributed. https://en.wikipedia.org/wiki/Distributional_semantics#Distributional_semantic_modeling_in_vector_spaces https://en.wikipedia.org/wiki/Distributional_semantics#Distr...
- Peteragain 11mo agoThis is the problem. I am arguing there are no distributed representations (cf Harnad's original symbol grounding problem paper, Hinton, and others). There are "distributional representations" by definition (cf the Wikipedia entry)
- srean 11mo agoIn the discussion thread there seems to be a somewhat contentious argument about what is a Markov Model, whether LLMs are one, RNNs are one and so on. Markov Models are anything that has state and emit tokens based only on its current state and undergoes a state transition. The token emission and state transitions are usually probabilistic -- a statistical/probabilistic analogue of a state machine. The deterministic state machine is a special case where the transition probabilities are degenerate (concentrated at an unique point). For a Markov Model to be non-vacuous, non-vapid discussion point, however, one needs to specify very precisely the relationships allowed between state and tokens/observations, whether it's hidden or visible, discrete or continuous, fixed context length or variable context length, causal or non causal ... The simplest such model is one where the state is a specified, computable function of the last k observations. One such simple function is the identity function -- the state then is the last k tokens. This is called a k order Markov Chain and is a restriction of the bigger class -- Markov Models. One can make the state a specified, computable function of (k) previous states and k most recent tokens/observations. (Equivalently RNNs) The functions may be specified only upto a class of computable functions, finite or infinite in size. They may be stochastic in the sense they define only the state transition probabilities. You can make the context length a computable function of the k most recent observations (therefore they can be of varying length), but you have to ensure that the contexts are always full for this model to be well defined. Context length can be a computable function of both the (el) most recent states and k most recent observations. Crazy ones emit more than one token based on current state. On and on. Not all Markov Models are learnable.
- frizlab 11mo ago> Also, these days, one hardly needs a Markov model to generate gibberish; social media provides an ample supply.
- anthk 11mo agoQuick test for Perl users (so anyone there with a Unix-like). Run these as a NON root user: cpanm -n local::lib cpanm -n Hailo ~/perl5/bin/hailo -E Scored -t corpus.txt -b brain.brn ~/perl5/bin/hailo -b brain.brn As corpus.txt, you can use a Perl/sed command for instance with book from Gutenberg. I forgot to put the '-E' flag in my previous comments, so here it is. It's to select a more 'complex' engine, so the text output looks less gibberish.
- msapaydin 11mo agohttps://cdn.cs50.net/ai/2023/x/lectures/6/src6/markov/# https://cdn.cs50.net/ai/2023/x/lectures/6/src6/markov/# This is a nice Markov text generator.
- litver 11mo ago"I Fed 24 Years of My Blog Posts to a Markov Model" you're not the first who did it. Already dozens of LLMs did it.
- pessimizer 11mo agoYou could literally buy this at Egghead software for $3 from the bargain bin in 1992. I know, because I did. I fed it 5 years worth of my juvenile rants, and laughed at how pompous I sounded through a blender. https://archive.org/details/Babble_1020 https://archive.org/details/Babble_1020 A fairly prescient example of how long ago 4 years was: https://forum.winworldpc.com/discussion/12953/software-spotlight-babble https://forum.winworldpc.com/discussion/12953/software-spotl...
- delfugal 11mo agoShould call it Trump Speech Generator. Loads of gibberish.
- rumgewieselt 11mo agoI love the design of the website more than the Markov model. Good Job!
- huflungdung 11mo ago[dead]
- keithalewis 11mo agohttps://math.uchicago.edu/~shmuel/Network-course-readings/MCMCRev.pdf https://math.uchicago.edu/~shmuel/Network-course-readings/MC...
- effnorwood 11mo agoIt laughed and gave him a kiss