3 ms·
Well, except that Kafka doesn't automatically delete stuff?
by skybrian 9y ago
Well, except that Kafka doesn't automatically delete stuff?
- cookiecaper 9y agohttps://kafka.apache.org/documentation/ https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data older and/or bigger than Y or Z" flag on a RDBMS, because such a flag doesn't exist. --- EDIT: Since my other post got a few downvotes, and my account has been sanctioned in the past for posting similar thoughts about tech-hipsters diving head-first into Kubernetes without understanding any of the effects of doing so, I can no longer post replies for a period of some hours. As such, I will copy my reply to child comment from snuxoll below, which I was not allowed to post, and then log out for a while. Kafka devs, seek investment from YC, you may get them to sanction accounts that are too critical of the inappropriate adoption of your project, as they've done to me for criticizing Docker/k8s. reply: >Log compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage). Yeah, I didn't explicitly detail the compaction option because it's still potentially deleting huge amounts of data. It promises to keep the latest state, which, yes, is better than deleting everything. But why not just not delete anything? The article says they're working with 100GB of data. There are VIDEO GAMES that get downloaded to consumer hard disks that occupy more than that (also not really excusably, but still). I understand why Kafka provides these semantics and I think it's great that they do. The issue is that developers choose to ignore them, and seriously imperil their data's well-being by doing so.
- dang 9y agoI'm not quite sure what you're insinuating about YC here but I suppose a standard "no, we don't do anything like that" is in order. We sometimes rate-limit HN accounts when they have a habit of posting low-quality comments too quickly or (especially) getting involved in flamewars. Since we've discussed this more than once before, I assume you remember it, but other people might not know. That's all that's happening here and of course it has nothing to do with your opinions, about Kafka or YC or anything else.
- cookiecaper 9y ago"Flame wars" in the sense of actually responding to the people who are trying to have a discussion on the technical points? Suggesting that my posts involve flaming virtually ever is a flagrant mischaracterization, and we both know it. This thread contains about the "worst" you'll find from me, with "No offense, but this reveals your ignorance...". This is a flame only insofar as "flame" constitutes any disagreement at all. When the account was rate-limited, the complaint was a) that HN had received out-of-band complaints from people who were getting mad that I discouraged others from deploying database clusters on Kubernetes; and b) that my posts on the subject were "trite". Maybe I don't waste enough time on HN, but I'd never seen that before. I concede there was one post in that thread that could be interpreted as borderline incendiary, suggesting that the only reason to run a database on top of k8s is to win "GCool Points" (a position I continue to maintain), but it was directed at no one in particular, intended to provide some levity to non-techno-hipsters, and prevent some of the routine noise we see every time I make that type of post from people who really don't have any counter-point except that "Google does it!". Hardly a habit of engaging in flame wars. And it was used by HN/YC as an excuse to "detach" the entire thread, in which a real technical debate was occurring, as is occurring here, instead of just the single borderline post. I understand that you feel the need to go on the record with a denial. I hope you can appreciate that I feel that need too, whether the accusation is implied through the rate limit that prevents me from replying and may mislead others into believing that my position is indefensible, or explicit, as it is now.
- snuxoll 9y agoLog compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage).
- LgWoodenBadger 9y agoYou may not have to worry about somebody setting a "delete all data older and/or bigger than Y or Z" but you have to worry about someone running "DELETE FROM table" without a WHERE clause. Which is easier to prevent? The one that can be done through the same mechanism as non-destructive queries? Or the one that can only be modified through a file-system configuration, completely separate from its API? Regardless, it's a different paradigm with different "don't do that" behaviors that you need to know about. In Kafka, if you want the persistent, append-only, write-ahead log to not delete stuff, then configure the retention period to keep things forever.
- cookiecaper 9y agoIf someone runs `DELETE FROM table` without a WHERE clause, I expect: a) the query to be tested and scripted on non-production environments first, making this a non-issue; b) user doesn't have DELETE permissions on that table and/or the rows not intended to be deleted; c) referential integrity to kick in and prevent the deletion of interdependent records (which is most records in a database); d) CHECK constraints, triggers, and other validation routines to prevent this clearly-excessive operation; e) the person executing and inspecting these queries within an ad-hoc transaction to roll it back before committing; f) if, in the event this does occur and commit, which itself means there's a big problem with your procedure, streaming binlog archives can facilitate a point-in-time backup, audit tables can be used to rebuild the data, etc.; these aren't typically included by default (streaming point-in-time backups are on AWS Aurora) but they're conventional for many professionally-run RDBMS installations. and I'm sure there are failsafes that I'm forgetting, and since I'm not a DBA, some I'm probably not even aware of. How many of these can I expect to help me out when a packaging bug (or, simply a mistaken "y" on the prompt asking if I want to override the package config) clobbers the Kafka config file?