3 ms·

I've worked for a while now in the field and I never worked with kafka. Nor most of the peers I've worked with. And I wouldn't even call it the most used messag

by chucke 4y ago

I've worked for a while now in the field and I never worked with kafka. Nor most of the peers I've worked with. And I wouldn't even call it the most used message queue or data stream solution, so I don't understand why people treat it as ubiquitous software. SQL, on the other hand, is. And while it's certainly not built for message queues, that's not the point of the article.

sjducb 4y ago
That's not my argument. I'm saying it'll be easier to onboard developers onto a standard message queue (like Kafka) than to onboard developers onto some custom thing you wrote. The original article is rebranding "Not invented here" as "Minimise dependencies"
- chucke 4y ago
  But kafka isn't a standard message queue, and therefore won't make onboarding easier except for the ones who already know it, which is not a significant majority, right? Or at least we'll have to agree on the definition of standard here. For the equivalents of SQL, I'd say we have AMQP or MQTT, and kafka does not seem to support any (my goggling tells me it has its own protocol); deployment-wise, it doesn't seem easier to set up than, let's say, SQS; while used and supported in most common languages, most queue/job frameworks do not seem to support it OOTB (looked into ruby, python, php, javascript), so you'd have to write your own custom kafka handling code from scratch. So what would make it easier than just riding postgres?
  - sjducb 4y ago
    For me standard means used by more than one company and well documented. It also means that most companies use it in the same way. Basically all 3rd party Frameworks and tools are "standard" Custom means that you wrote a unique solution for yourself. No-one else uses your solution. New hires have to learn it from scratch either by reading your code or your own custom documentation. There is obviously more work to onboard people onto a custom solution. You can hire people who understand the standard solutions. Postgres isn't a queue or a message broker. You are writing your own queue that depends on postgres. Your custom queue will have complexity that new hires have to learn. No new hire will have used your custom in house queuing system before. If you want to use SQS that's a great idea. I support using standard solutions.

by chucke 4y ago

sjducb 4y ago

That's not my argument. I'm saying it'll be easier to onboard developers onto a standard message queue (like Kafka) than to onboard developers onto some custom thing you wrote. The original article is rebranding "Not invented here" as "Minimise dependencies"

chucke 4y ago
But kafka isn't a standard message queue, and therefore won't make onboarding easier except for the ones who already know it, which is not a significant majority, right? Or at least we'll have to agree on the definition of standard here. For the equivalents of SQL, I'd say we have AMQP or MQTT, and kafka does not seem to support any (my goggling tells me it has its own protocol); deployment-wise, it doesn't seem easier to set up than, let's say, SQS; while used and supported in most common languages, most queue/job frameworks do not seem to support it OOTB (looked into ruby, python, php, javascript), so you'd have to write your own custom kafka handling code from scratch. So what would make it easier than just riding postgres?
- sjducb 4y ago
  For me standard means used by more than one company and well documented. It also means that most companies use it in the same way. Basically all 3rd party Frameworks and tools are "standard" Custom means that you wrote a unique solution for yourself. No-one else uses your solution. New hires have to learn it from scratch either by reading your code or your own custom documentation. There is obviously more work to onboard people onto a custom solution. You can hire people who understand the standard solutions. Postgres isn't a queue or a message broker. You are writing your own queue that depends on postgres. Your custom queue will have complexity that new hires have to learn. No new hire will have used your custom in house queuing system before. If you want to use SQS that's a great idea. I support using standard solutions.