3 ms·

The general trend in these posts (and tweet storms) is the same - pick any technology and then preach the wisdom of not using it because "You're not Google", or

The general trend in these posts (and tweet storms) is the same - pick any technology and then preach the wisdom of not using it because "You're not Google", or "You won't need it". It's a great tactic as it puts the speaker in a position of judgement, and they can freely share their opinions on some architecture, without getting any pushback from the creators of said architecture. Quite a one-sided affair that makes you look like you know everything. It's, frankly, elitist. This entire article can be surmised by "You don't need Kafka because the throughput seems to be low". To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka. But of course, "You don't need Kafka" because <reasons>.

hyperpape 6y ago
> To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka. WeWork has very predictable scale because its data ingestion needs are tied to expensive investments in physical capital. They don't have to worry about getting slashdotted. Same thing at my company, where data needs are tied to expensive enterprise contracts. In fact, we're considering Kafka for something we do, but it didn't sneak up on us--we knew for years that a particular use case involved a lot of data and wasn't ideal for a relational database.
kumarvvr 6y ago
> It's, frankly, elitist Why not see the coin from the other side. Developers giving their experience when they are working sane ? Many of these articles, I have seen authors quoting from their experience building large systems. Moreover, as a system architect, you often have to analyze the requirements and choose a good stack for implementation. A young dev in a startup is biased against the architecture in one way, because they might be seeing Kafka on their resume as a shining star to greater riches.
- manish_gill 6y ago
  The interesting part is that indeed, the people who write these articles are usually building large scale systems, writing books, newsletters and building companies for them, and then telling everyone else how they don't need it. Of course engineers have a tendency to over-engineer solutions. I don't think anyone is surprised by that. But this "oh you don't need it, I've used it at <bigcorp>, and you're not that" thinking just rubs me wrong. Especially when it's backed up by hollow arguments like the ones in this post.

hyperpape 6y ago

> To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka. WeWork has very predictable scale because its data ingestion needs are tied to expensive investments in physical capital. They don't have to worry about getting slashdotted. Same thing at my company, where data needs are tied to expensive enterprise contracts. In fact, we're considering Kafka for something we do, but it didn't sneak up on us--we knew for years that a particular use case involved a lot of data and wasn't ideal for a relational database.

kumarvvr 6y ago

> It's, frankly, elitist Why not see the coin from the other side. Developers giving their experience when they are working sane ? Many of these articles, I have seen authors quoting from their experience building large systems. Moreover, as a system architect, you often have to analyze the requirements and choose a good stack for implementation. A young dev in a startup is biased against the architecture in one way, because they might be seeing Kafka on their resume as a shining star to greater riches.

manish_gill 6y ago
The interesting part is that indeed, the people who write these articles are usually building large scale systems, writing books, newsletters and building companies for them, and then telling everyone else how they don't need it. Of course engineers have a tendency to over-engineer solutions. I don't think anyone is surprised by that. But this "oh you don't need it, I've used it at <bigcorp>, and you're not that" thinking just rubs me wrong. Especially when it's backed up by hollow arguments like the ones in this post.