8 ms·
Understanding Kafka with Factorio
- DonHopkins 7y agoWhen I saw this title, I first thought it was about understanding the cockroach-like aliens in Factorio, but it is actually about queuing and congestion and parallel processing with conveyor belts and factories! With the caveat that Factorio is EXTREMELY addictive, properly described as "programmer crack", here are some other Factorio related discussions: Factorio – a game where you can automate basically anything (factorio.com) https://news.ycombinator.com/item?id=11266471 https://news.ycombinator.com/item?id=11266471 http://www.factorio.com/ http://www.factorio.com/ "This game is like crack for programmers." -kentonv (Tech lead for Cloudflare Workers, author of Sandstorm.io, Cap'n Proto) https://news.ycombinator.com/item?id=11269098 https://news.ycombinator.com/item?id=11269098 Raycasting engine in Factorio game – Facto-RayO v1.0 [video] (youtube.com) https://news.ycombinator.com/item?id=19878688 https://news.ycombinator.com/item?id=19878688 https://www.youtube.com/watch?v=7lVAFcDX4eM https://www.youtube.com/watch?v=7lVAFcDX4eM Factorio to talk about horizontal scaling of software services (medium.com) https://news.ycombinator.com/item?id=19095271 https://news.ycombinator.com/item?id=19095271 https://medium.com/@chrisfauerbach/gamification-of-horizontal-scaling-4a6c394c9e75 https://medium.com/@chrisfauerbach/gamification-of-horizonta... Mapping Factorio with Leaflet (cloudflare.com) https://news.ycombinator.com/item?id=18185058 https://news.ycombinator.com/item?id=18185058 https://blog.cloudflare.com/mapping-factorio-with-leaflet/ https://blog.cloudflare.com/mapping-factorio-with-leaflet/
- TeMPOraL 7y agoSome random Factorio-themed things of my own: https://imgur.com/c51pTnP https://imgur.com/c51pTnP - here I tried to invent a way to solve the accessibility issues of moving walkways; https://news.ycombinator.com/item?id=17703131 https://news.ycombinator.com/item?id=17703131 for context. https://mastodon.technology/@temporal/100646861775747986 https://mastodon.technology/@temporal/100646861775747986 - Factorio-inspired tootstorm about what I'd like to see in an IDE/debugger.
- X6S1x6Okd1st 7y agoPretty off topic, but I'd appreciate it if you didn't use "____ crack" as a way to describe something that is addictive.
- dEnigma 7y agoWhy? Isn't crack famously (or infamously) addictive?
- klyrs 7y agoMy understanding is that it's similar to people taking offense at the term "grammar nazi" because it trivializes an atrocity
- ZeroGravitas 7y agoI agree with the (apparently unpopular) notion and I avoid describing things as "x crack" and wince a bit when people use it as it seems a bit tone-deaf and childish, like referring to something you don't like that there's too much of as "programmer AIDS" or a bad design as a "UI Holocaust" or a failed project as an "abortion" or a bad sports defeat as being "raped", all of which make a similar kind of allegorical sense but are more obviously tacky. For whatever reason crack allusions are a bad taste joke that people (unlike most of the above) don't seem to realise is bad taste, possibly because they consider crack addicts as either other or lesser to them, rather than unfortunate victims with a shared humanity.
- scotty79 7y agoI think that kind of word use is inevitable and actually a sign of something good. It for me shows that people know about these things but they are percieved to be so unbelievably horrible that they feel unreal and as such can serve as metaphors. I know these things happened, happen and most likely will happen and such metaphors are unpleasant for people affected, but I think it's natural language evolution. People still get murdered occasionally and families of murder victims might be uneasy when they hear about someone that "he killed her with this joke!" but it won't prevent people from using that phrase because murders are so rare for most people that the word gradually loses its literal meaning.
- saint-loup 7y agoOh my, can someone ping Gwern to add Factorio to this list. https://www.gwern.net/Turing-complete https://www.gwern.net/Turing-complete
- GauntletWizard 7y agoThere's neither surprise nor accident in Factorio's Turing completeness. The developers were aware of and fans of the existence of proofs of games on that list (Transport Tycoon is a particular example that they intentionally modeled train mechanics from)
- Baeocystin 7y agoPro-tip to others who think 'Programmer Crack' is hyperbole- it is not. Factorio does have a rather steep learning curve, and the UI is not the most intuitive. But. Once how it works clicks, it clicks, and suddenly morning. And I'm in my 40's, I'd thought my days of unexpected overnights while working on fascinating problems were mostly behind me. Highly recommended, but be aware.
- mabbo 7y agoI had to uninstall it last weekend because I came to realize my playing it was no longer healthy. I mean, that sounds funny, but I'm being totally serious. I've had a few games like that in my life, and I've come to recognize a few signs that what I'm doing is not good for me anymore. But wow, what a great game.
- GaryNumanVevo 7y agoWhenever they release a new version, I'll boot it up and waste the better part of a weekend to get another mega-base running. Always satisfying
- jjjensen90 7y agoI've only had 2 games like that in my life, EverQuest (also called EverCrack) and Factorio (also called Programmer Crack). I had to go cold turkey or I was skipping the gym and meals and not sleeping!
- Baeocystin 7y agoOne thing I credit Factorio for (and I am not joking at all when I say this, despite how ridiculous the statement might seem) is learning to be a little more understanding about folks that deal with addictions. I've never had a problem not doing something to excess, so I never really understood people who do. But when I first discovered Factorio a few years back, there were enough times where I didn't do $ActuallyRatherImportantThingX because I was allllmost done with some aspect of a megabase to give me pause. There are lots of games I can play for a specified amount of time and be fine with it. Pretty much all of them, really. Factorio, I get annoyed if pulled away in the middle of a project. So now I know to not even start unless it's free time for the rest of the night.
- gnulinux 7y agoFor those who don't know factorio is truly a masterpiece of a game. It's described as "programming the game". Strongly recommended to everyone here.
- briffle 7y ago"Oh, s##t, I have to wake up in 2 hours" -- said at least once by anyone who has played it...
- gnulinux 7y agoOh man, all those hours wasted trying to get bots so that I can automate iron outpost building only to get destroyed by a huge wave of biters. Epic memories.
- root_axis 7y agoAnd high quality network multiplayer! "Pair programming" was never this much fun!
- drivers99 7y agoI used to play this with co-workers and we would end up having a standup meeting about it once we got back to the office in the morning.
- DonHopkins 7y agoKatherine of Sky has some great "pair programming" videos where she teams up with other experienced players. It's amazing to see how diverse different people's styles can be! You really get insight into how their minds works. I can't remember which ones are multi player, but here are some of her playlists -- awesome stuff: Factorio Tutorials Playlist https://www.youtube.com/playlist?list=PL4o6UvJIdPNpm9szThfI-2y3Bv0c3rPlB https://www.youtube.com/playlist?list=PL4o6UvJIdPNpm9szThfI-... Factorio 0.17: Entry Level to Megabase 2 https://www.youtube.com/watch?v=el9Ih1gg3i4&list=PL4o6UvJIdPNpTpKrROWTePBP1VyYBfRAD https://www.youtube.com/watch?v=el9Ih1gg3i4&list=PL4o6UvJIdP... Factorio 0.17: Pyanodon with Aven1017 https://www.youtube.com/playlist?list=PL4o6UvJIdPNpUZ230345GnEs2JbGVJkGX https://www.youtube.com/playlist?list=PL4o6UvJIdPNpUZ230345G... Factorio Landfill Avalanche! (MP with Aven!) https://www.youtube.com/playlist?list=PL4o6UvJIdPNrEWhjZOzfC5eOHPJiKS36B https://www.youtube.com/playlist?list=PL4o6UvJIdPNrEWhjZOzfC... Factorio ASMR https://www.youtube.com/watch?v=P7dasB1a5J0&list=PL4o6UvJIdPNrRBRcBl2u5_yZ5sOFJPTsE https://www.youtube.com/watch?v=P7dasB1a5J0&list=PL4o6UvJIdP... >Rabbi Tricky: OMG KoS, your voice, your choice of words, it makes me really relaxed. I saw it mentioned before, but its like Bob Ross. Happy little trees are now in my mind, together with happy little miners and such. Please don't make this a one time video, i really love this. Much love >mithrane: KOS - You really are the Female Bob Ross of Factorio. Love your videos and love this concept.
- realityman 7y agoA better way to understand Kafka is to fill a room with rakes, turn of the lights, and walk around.
- jpk 7y agoThis sounds like an idea founded on experience, but without including any details about how you arrived at this sentiment the comment doesn't bring any value to the thread.
- breischl 7y agoNot GP, but early versions of Kafka had a lot of poorly-chosen default settings and sharp edges. eg, it was tuned for speed and availability over reliability, so it could lose your data if you weren't careful (not dissimilar to early versions of MongoDB that way). It was possible/easy to set up topics with no replication, so you'd lose data on your first server outage. KStreams had no real error handling - it would just explode and not even really tell you why. I wasn't as close to the operational side, but there were a lot of problems there too. IME it has gotten better over time, but it's still pretty easy to shoot yourself in the foot if you don't understand what you're doing with it.
- DonHopkins 7y agoDoesn't Kafka also support mouse traps, too?
- amdelamar 7y agoI think the analogy works better with queues and distributed services in general, rather than just Kafka.
- chrischen 7y agoThe analogy helped me understand, but in factorio belts are required primarily because everything takes physical space. What are real world use cases for kafka? Is it primarily if your data is too much to simply store in a db?
- penagwin 7y ago> What are real world use cases for kafka? He covers it under the introduction "Why bother with async messaging?". Basically in the analogy the belts are queues. Factorio doesn't require queues, but it quickly leads to problems with resource management, as you essentially have to lock the system until your item is done processing.
- pdpi 7y agoPush architectures put all of the control on the producer, but become quite painful in the face of an unreliable consumer. Pull architectures put all of the control on the consumer, but become quite painful in the face of an unreliable producer. Message queues allow the producer to think it's participating in a push architecture, while the consumer believes it's a pull-based arch. Because the message queue itself does very little work, it gets to play a very well-behaved consumer insofar as the producer is concerned, while behaving like a well-behaved producer for the consumer. The queue's buffering then allows both the producer and the consumer to misbehave to some degree while the queue itself keeps the illusion of good behaviour. Kafka specifically falls under the category of log-oriented message queues, which are eminently useful for distributing any workload that looks like "tail a log and process each line as it comes in" across a large number of nodes.
- pul 7y agoThis. With the slight nuance that it works better if you don't think of it as queues, since that suggests messages being removed when processed. The big win when introducing Kafka is that (apart from the schema) it completely decouples producers and consumers of messages. That in turn reduces the required coordination between independent teams within an organisation.
- 7y ago
- pul 7y agoI wrote this article a couple of months ago and didn't expect it to appear on my news feed all of the sudden. Thanks for the love everyone. Any questions are welcome of course!
- Baeocystin 7y agoHow do you like the .17 update? :D More seriously, I am curious about what people who haven't played Factorio think of the article. Since I've sunk $bignum hours in to it, I found your explanations clear, but I'd like to hear what those who haven't think.
- IggleSniggle 7y agoI haven’t played Factorio, and have just barely used Kafka. Therefore, if there was any subtlety, it was lost on me. Nonetheless, this article was clear. I thought this cute and I liked it, but I’m not sure that I gained any new insights into Kafka. This is not a criticism! However, one of the main advantages of Kafka, I thought, was that it’s so much more than a queue, since data is replayable, etc. The article does mention this as a shortcoming of the analogy, but isn’t this one of the main selling points vs other message queues?
- jand 7y agoI really enjoyed reading this, thank you very much. Please check the picture with the label "All microservice instances consume all messages". From my understanding it is more of a depiction of a consumer group. As factorio really consumes the messages (they are gone once consumed) you cannot really implement "All microservice instances consume all messages". Maybe this is BS, i have no Kafka expertise.
- pul 7y agoYou're absolutely right. There's no correct way to visualize this in Factorio.
- Lowkeyloki 7y agoYou lost me at the part about partitions. It might help if those gifs were labeled.
- pul 7y agoYeah, I started out labeling them, but that turned out to be extremely tedious. Does anyone know of tools that make annotating gifs easy?
- rs23296008n1 7y agoIsn't this a repost?
- DonHopkins 7y agoDang suggested reposting it since it didn't get much attention the first time.
- rs23296008n1 7y agoWow, I ask a valid question about a old post and I'm punished with multiple downvotes. Lesson learnt.
- NullPrefix 7y agoYou were downvoted because you did not RTF FAQ.
- rs23296008n1 7y agoBut you're happy to abuse me. So that is acceptable then.
- NullPrefix 7y agoHow did you came to the conclusion that I am abusing you by pointing out that you did not read the FAQ? My message was not even unsolicited. You were asking why and I gave you an answer.
- fragmede 7y agoThere's a similar web game - http://www.factoryidle.com http://www.factoryidle.com, for those who can't/won't play FactorIO, with a similarly addictive quality.
- allannienhuis 7y agoI'm getting 403 bad request responses from cloudfront on that url
- xmonkee 7y agoThat's crazy! I installed it yesterday for the first time to try out the steam proton thing and was literally up till 3 am. I didn't realise the programmer crack angle till I read this. I'm screwed.
- glandium 7y agoThere's a Linux native version, no need for Proton.
- Itsdijital 7y agoYou're screwed. Also press Alt And use "Q" while hovering over an object to put it in hand (if it's in your inventory). Press "Q" again to drop whats in hand. I played waaayyy to long without knowing this and it is unbelievably useful, saves mountains of time digging through your inventory.
- Baeocystin 7y agoAlso, Z to drop something on the ground if you find your bots a little to assertive about filling your inventory back up and you find yourself stuck. :D
- DonHopkins 7y agoOh no, what have I wrought? Another addict. Well hopefully it served as a warning to more people than the number of people it tempted into getting hooked. It’s only because I haven’t set up my PC since I moved that I have had any time in my life not playing Factorio myself. I still have the dreams, though.