12 ms·
Understanding Kafka with Factorio (2019)
- morelisp 3y agoCute, but over years of explaining it I think any explanation of Kafka that presents it as a queue is bound to leave the reader with more misaligned expectations than when they started (while also making them think they learned something, which can be even more dangerous). To keep the Factorio-esque framing, move the consumers, not the messages.
- kentm 3y agoAgreed. There's an important difference between things like Kafka & Kinesis vs RabbitMQ & SQS. The latter are conceptually queues, and the former are conceptually logs. Logs and queues can both be used in many of the same use cases, but it's important to understand how they are different.
- Terr_ 3y agoi.e.: Items are frequently removed from queues which have a shared "next item", while logs usually just get longer and each consumer is responsible for keeping track of their own progress or positions. It's harder to think of factory-game analogies for logs, since they involve copying without altering the original sequence. It would have to involve some kind of moving non-destructive sensor or object-cloner mechanic.
- marginalia_nu 3y ago> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.
- geodel 3y agoThe idea is don't let the logic come in the way of promoting "web scale" software.
- morelisp 3y agoThe kind of server you'd run Kafka on tends to already be pretty far up the curve. I don't think I can get 10x our default broker for 20x the cost. Maybe 100x the cost. (I could probably get 2x it for 2x the cost but once you value HA the practical inflection point starts below the actual cost intersection.)
- teawrecks 3y agoFor small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?
- Sohcahtoa82 3y agoThink cores instead of clock speeds. In the case of cloud instances, doubling cores is frequently less than 100% more expensive.
- The_Colonel 3y agoIncreasing core count is not really vertical scaling. It's a hybrid between vertical and horizontal scaling, having some characteristics of both. It also tops out quite early (especially its cost-effectiveness for many use cases, but there's an absolute upper limit as well).
- morelisp 3y agohttps://aws.amazon.com/msk/pricing/ https://aws.amazon.com/msk/pricing/ prices scale linearly with CPU beginning with m5.large, and I wouldn't really want to run a production Kafka on anything less than m5.xlarge. (They do at least keep linearly scaling all the way up.) Speculating wildly, I could probably have run some of our real clusters on the equivalent of a 8xlarge, but of course 32 core systems were not widely available at that time. The cluster I run today, even a hypothetical 48xlarge would struggle. YMMV for non-managed stuff, but really, you can only bump cores like 3 times realistically, 4 if you started really shitty, before you start getting into special pricing brackets.
- BSEdlMMldESB 3y agowhat about understanding our own imperialistic civilization? crash in planet (or new continent?) and proceed to expand while decimating native life? in any case, good game.
- golergka 3y agoThat's what all life forms do — consume everything they can and reproduce as much as they can. So, doing is _is_ life.
- hutzlibu 3y agoThat's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.
- golergka 3y agoThose statements do not contradict each other. Life forms consume and reproduce as much as they can, and they also live in cooperation and symbiosis. The latter can often be the most effective strategy to consume and reproduce as much as one wants.
- BSEdlMMldESB 3y agoyes, agree the issue then, is how to decide what "as much as we can" even means; if we do "as much as we can" but this kills us, then it will have turned out that we indeed could not do that much
- hutzlibu 3y ago"That's what all life forms do — consume everything they can and reproduce as much as they can" Well, if you take this statement literal, then I think they do contradict each other - see my cancer example. But in the way you apparently meant it, then no, there is no contradiction.
- jrm4 3y agoI was 100% expecting something about the writer. :)
- geodel 3y agoBad software is way more popular than a good writer.
- jrm4 3y agoWhy are you downvoting him, hes right? :)
- lukasLansky 3y agoBoth Franz Kafka and Factorio come from Prague after all.
- golergka 3y agoOne morning, when Gregor Samsa woke from troubled dreams, he found himself transformed into a piece of iron ore on a moving conveyor belt.
- tobbe2064 3y agoThat one mad me laugh so much i annoyed my wife
- dang 3y agoRelated: Understanding Kafka with Factorio (2019) - https://news.ycombinator.com/item?id=29304414 https://news.ycombinator.com/item?id=29304414 - Nov 2021 (72 comments) Understanding Kafka with Factorio - https://news.ycombinator.com/item?id=20362179 https://news.ycombinator.com/item?id=20362179 - July 2019 (84 comments) (Reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)
- cubefox 3y agoSemi related: Are other people also annoyed by how many projects are using names of completely unrelated famous things? I expected to read some wild association between a game and Franz Kafka, but no, it's about a streaming platform which happens to be named "Kafka". This is getting seriously annoying when you google for some, e.g. historical, term and then your search results are littered by some completely unrelated software/IT project which reuses the name for no reason in particular. "Factorio" is actually an example of how to do better: Just make up your own word!
- the_af 3y agoEven worse, Kafka and Apache Kafka have almost no meaningful connection. According to Wikipedia, the author was reading (Franz) Kafka, who was a writer, and the software system is "optimized for writing". (Franz) Kafka wasn't "optimized" for anything, so this is just whimsical naming. It could just as well have been named Apache Hemingway, or Apache Tolstoy. Whimsical naming is ok, but can also be confusing and annoying.
- cubefox 3y agoI think many such names are mostly chosen because they kind of sound nice and they have a prestigious association.
- titanomachy 3y agoWow, that's so unrelated. I think I'd tacitly assumed that it was named that because adopting an event-driven architecture results in byzantine and overly complicated software, like the bureaucracies in Kafka's novels.
- bee_rider 3y agoI’d be shocked if your explanation wasn’t the real one. “Optimized for writing” sounds like the sort of justification you give when your project with a sarcastic self-deprecating name becomes surprisingly successful. If they wanted to name it after a fast, efficient famous writer it would be Apache Hemingway. Kafka died before most of his writing was published and most of it was destroyed, which doesn’t seem like something the software would want to be associated with, right?
- not_your_mentat 3y agoAnd now I think of everything I do in Factorio models.
- AbraKdabra 3y agoIt's always good to se Cracktorio on the frontpage of HN, hopefully someone will make a similar article showing the similarities between Factorio and drugs.
- mistermann 3y agoOr human culture as a video game in general, and humans as semi-rational, semi-aware characters in the video game, except some of the characters are special in that their job is to deceive and exploit other characters, including constructing illusions indistinguishable from reality of who is good and who is bad, what is true and what is not, what we should do and should not do, etc. And to make it all even more exciting, not all of the illusionists are aware of their actual, often hybrid role in the big scheme of things. Maybe Shakespeare or some famous philosophers would have seen this angle, were video games to exist in their era. Unfortunate timing I guess.
- mavu 3y agoI clicked this and was immeasurably disapointed that the article talks about Apache Kafka, and not the author Kafka and how to understand his work with Factorio. That would be a much much better article.
- politician 3y ago[flagged]
- RhodesianHunter 3y agoI'm not sure how to feel about the fact that we're starting to see ChatGPT responses to questions like this in forums.
- mavu 3y agoI'm willing to make an exception in this case, because using LLMs for nonsense is exactly the usecase they are built for.
- lionkor 3y agoYes, but nonesense has no place here. There are other places to shitpost, namely this would be perfectly placed on the r/copypasta subreddit (NSFW, navigate with caution)
- politician 3y agoIt's getting too troublesome to post to HN. Three levels deep of self-affirming comments about why the original post should be censored. Cool.
- lionkor 3y ago[removed for opinion]
- haswell 3y agoI recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the program, but rarely can the user make sense of what's going on by seeing stuff moving around as the thing executes. And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain.
- gjulianm 3y ago> And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain. I feel the same. It scratches so many itches. I made the mistake of installing Space Exploration mod after the first run… so many hours invested now.
- lsaferite 3y agoSE is such a deep hole to fall into. I feel for you.
- swyx 3y agoany good explainers of whats fun about SE for those who will never play it?
- fardo 3y agoIt adds hundreds of hours of gameplay, new endings, multiple planets, and much longer progression chains that really push your ability to optimize to the limit. If the base game isn’t enough to keep you interested anymore because it’s gotten too simple, space exploration is a good hand grenade to mix things up.
- deleted 3y ago[deleted]
- lakomen 3y agoIdk... maybe it's because I'm self taught and have been coding since the age of 11, but I don't find the indirect approach helpful, the opposite. I believe that's why OO is so popular, people who only know the object way of thinking, who have difficulties with the virtual and abstract like OO and condemn the pragmatic approach.
- alexleeds 3y agoOK. 3rd repost of the same material. Really?
- aledalgrande 3y ago> If you don’t have a lot of time to spare, don’t download Factorio. Wish I read this months ago. Factorio sucked my life in until I managed to launch that damn ship
- henrydark 3y agoWho else was disappointed to find it's Apache Kafka and not Franz Kafka?