3 ms·

Hi everyone, I hope you like my latest side project! I'm an astronomy student who likes programming in his free time. This time I wanted to write something tha

Hi everyone, I hope you like my latest side project! I'm an astronomy student who likes programming in his free time. This time I wanted to write something that handles larger amounts of data. And as I recently came across the Stack Exchange data dump including all questions and answers I had the idea of using them to create Markov Chains for (nearly) every Stack Exchange site. The website displays the resulting content, which is often surprisingly coherent and entertaing, and allows upvotes/downvotes so that the best questions get to the front page. And as a bonus, I created a quiz where one can guess which site a random question is based on. If you are interested in my other projects, check out https://lw1.at https://lw1.at, if you want to see the code, everything is Open Source and can be found here: https://github.com/Findus23/se-simulator https://github.com/Findus23/se-simulator Please excuse the very minimal design, but after writing a lot of Single Page Applications I wanted to go the oposite way and write a website with less than 25KB.

reificator 8y ago
I'm not going to lie, I started reading this thinking it was an example. I was very impressed as often Markov chain generated sentences break down after a common joining word.
- SOLAR_FIELDS 8y ago
  I think it works better than other examples simply because your typical StackOverflow question is more often than not some fragmented sentence missing a few words which are then intended to be inferred from the body of the question. I suspect this has to do with the question length limit.
jaytaylor 8y ago
Really awesome that you shared the source code, I'm having a hard time trying to figure out how to get it all working, though. Any chance you'll be including setup instructions?
- Findus23 8y ago
  You are right, that's the problem when trying to get something in the world quickly instead of doing proper documentation. But I'll write something together today and send you a mail when I'm done.
  - jaytaylor 8y ago
    I see you've added setup instructions to the README. Thanks, you rock!
froindt 8y ago
>Please excuse the very minimal design, but after writing a lot of Single Page Applications I wanted to go the oposite way and write a website with less than 25KB. The world could use more of this. I took the simple quiz and my first question was from the Russian language site. I hadn't considered that Markov chains could work in other languages too. I wonder if there are any significant differences?
- Findus23 8y ago
  Hi, sorry everyone for the delay in answering, but timezones weren't nice to me. I really like minimalist pages, but I would be lying if I said the complete oposite (like https://lw1.at https://lw1.at) wasn't also fun to program. The whole text-generation is language-independent [1]. I just split the source up in words (well technically NLTK-tokens) and merge them together after generating the text. Therefore it even kind of works for Japanese [2] or Arabic. It's just a bit ugly as a lot of questions on non-English sites are in English and the chains end up as a mix. [1] https://github.com/Findus23/se-simulator/blob/master/markov.py https://github.com/Findus23/se-simulator/blob/master/markov.... and https://github.com/Findus23/se-simulator/blob/master/text_generator.py https://github.com/Findus23/se-simulator/blob/master/text_ge... [2] https://se-simulator.lw1.at/s/japanese.stackexchange.com https://se-simulator.lw1.at/s/japanese.stackexchange.com

reificator 8y ago

I'm not going to lie, I started reading this thinking it was an example. I was very impressed as often Markov chain generated sentences break down after a common joining word.

SOLAR_FIELDS 8y ago
I think it works better than other examples simply because your typical StackOverflow question is more often than not some fragmented sentence missing a few words which are then intended to be inferred from the body of the question. I suspect this has to do with the question length limit.

jaytaylor 8y ago

Really awesome that you shared the source code, I'm having a hard time trying to figure out how to get it all working, though. Any chance you'll be including setup instructions?

Findus23 8y ago
You are right, that's the problem when trying to get something in the world quickly instead of doing proper documentation. But I'll write something together today and send you a mail when I'm done.
- jaytaylor 8y ago
  I see you've added setup instructions to the README. Thanks, you rock!

froindt 8y ago

>Please excuse the very minimal design, but after writing a lot of Single Page Applications I wanted to go the oposite way and write a website with less than 25KB. The world could use more of this. I took the simple quiz and my first question was from the Russian language site. I hadn't considered that Markov chains could work in other languages too. I wonder if there are any significant differences?

Findus23 8y ago
Hi, sorry everyone for the delay in answering, but timezones weren't nice to me. I really like minimalist pages, but I would be lying if I said the complete oposite (like https://lw1.at https://lw1.at) wasn't also fun to program. The whole text-generation is language-independent [1]. I just split the source up in words (well technically NLTK-tokens) and merge them together after generating the text. Therefore it even kind of works for Japanese [2] or Arabic. It's just a bit ugly as a lot of questions on non-English sites are in English and the chains end up as a mix. [1] https://github.com/Findus23/se-simulator/blob/master/markov.py https://github.com/Findus23/se-simulator/blob/master/markov.... and https://github.com/Findus23/se-simulator/blob/master/text_generator.py https://github.com/Findus23/se-simulator/blob/master/text_ge... [2] https://se-simulator.lw1.at/s/japanese.stackexchange.com https://se-simulator.lw1.at/s/japanese.stackexchange.com