6 ms·

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is understated because these experimental setups misinterpret the functionality of the new architectures/formats they're using. It doesn't truly rear its ugly head until there is a major data loss or corruption event. It's not that these never happen with RDBMS, it's just that RDBMS contemplates this possibility and tries to make it pretty hard to do that, whereas message queues just automatically delete stuff (by design, so they can serve as functional message queues!). RDBMS have spoiled us and we take its featureset, 40+ years in the making, for granted. We need to be careful and not assume that `GROUP BY` is the only thing we leave on the table when we "adopt" (more accurately abuse) one of these new-wave solutions as a system of record. Since no one is going to admit to their boss "this wouldn't have happened if we used Postgres", and since most bosses are not going to know what that means, most of these spectacular failures will never be accurately attributed to their true cause: developers putting their interest in trying new things above their duty to ensure their employer's systems are reliable, stable, and resilient.

awinder 9y ago
There are non-negligible problems in the news space like: 1. Supporting full-text search for a fair number of concurrent users 2. Availability of the system with minimal downtime 3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic 4. Notifications I could go on and on but honestly, it's just a tone-deaf response. Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"
- cookiecaper 9y ago
  "We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."
skybrian 9y ago
Well, except that Kafka doesn't automatically delete stuff?
- cookiecaper 9y ago
  https://kafka.apache.org/documentation/ https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data older and/or bigger than Y or Z" flag on a RDBMS, because such a flag doesn't exist. --- EDIT: Since my other post got a few downvotes, and my account has been sanctioned in the past for posting similar thoughts about tech-hipsters diving head-first into Kubernetes without understanding any of the effects of doing so, I can no longer post replies for a period of some hours. As such, I will copy my reply to child comment from snuxoll below, which I was not allowed to post, and then log out for a while. Kafka devs, seek investment from YC, you may get them to sanction accounts that are too critical of the inappropriate adoption of your project, as they've done to me for criticizing Docker/k8s. reply: >Log compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage). Yeah, I didn't explicitly detail the compaction option because it's still potentially deleting huge amounts of data. It promises to keep the latest state, which, yes, is better than deleting everything. But why not just not delete anything? The article says they're working with 100GB of data. There are VIDEO GAMES that get downloaded to consumer hard disks that occupy more than that (also not really excusably, but still). I understand why Kafka provides these semantics and I think it's great that they do. The issue is that developers choose to ignore them, and seriously imperil their data's well-being by doing so.
  - dang 9y ago
    I'm not quite sure what you're insinuating about YC here but I suppose a standard "no, we don't do anything like that" is in order. We sometimes rate-limit HN accounts when they have a habit of posting low-quality comments too quickly or (especially) getting involved in flamewars. Since we've discussed this more than once before, I assume you remember it, but other people might not know. That's all that's happening here and of course it has nothing to do with your opinions, about Kafka or YC or anything else.

awinder 9y ago

There are non-negligible problems in the news space like: 1. Supporting full-text search for a fair number of concurrent users 2. Availability of the system with minimal downtime 3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic 4. Notifications I could go on and on but honestly, it's just a tone-deaf response. Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"

cookiecaper 9y ago
"We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."

skybrian 9y ago

Well, except that Kafka doesn't automatically delete stuff?

cookiecaper 9y ago
https://kafka.apache.org/documentation/ https://kafka.apache.org/documentation/ Name: cleanup.policy Description: A string that is either "delete" or "compact". This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. ??? How is this not automatic deletion of stuff? I don't have to worry about someone setting a "delete all data older and/or bigger than Y or Z" flag on a RDBMS, because such a flag doesn't exist. --- EDIT: Since my other post got a few downvotes, and my account has been sanctioned in the past for posting similar thoughts about tech-hipsters diving head-first into Kubernetes without understanding any of the effects of doing so, I can no longer post replies for a period of some hours. As such, I will copy my reply to child comment from snuxoll below, which I was not allowed to post, and then log out for a while. Kafka devs, seek investment from YC, you may get them to sanction accounts that are too critical of the inappropriate adoption of your project, as they've done to me for criticizing Docker/k8s. reply: >Log compaction is generally what you want, which will preserve the most recent message for every key in a topic. Event streams expanding boundlessly is something very few will ever need or want, so you'll toss messages into an "event" topic of some kind, apply the event to the most recent entry in the "model" topic and store a new version (which will be kept after log compaction, the original messages can be pruned after you need to free up storage). Yeah, I didn't explicitly detail the compaction option because it's still potentially deleting huge amounts of data. It promises to keep the latest state, which, yes, is better than deleting everything. But why not just not delete anything? The article says they're working with 100GB of data. There are VIDEO GAMES that get downloaded to consumer hard disks that occupy more than that (also not really excusably, but still). I understand why Kafka provides these semantics and I think it's great that they do. The issue is that developers choose to ignore them, and seriously imperil their data's well-being by doing so.
- dang 9y ago
  I'm not quite sure what you're insinuating about YC here but I suppose a standard "no, we don't do anything like that" is in order. We sometimes rate-limit HN accounts when they have a habit of posting low-quality comments too quickly or (especially) getting involved in flamewars. Since we've discussed this more than once before, I assume you remember it, but other people might not know. That's all that's happening here and of course it has nothing to do with your opinions, about Kafka or YC or anything else.