2 ms·

I have at a few companies now. It's great for processing data that goes beyond a single database call, data formatting and presenting something on a page. If

by monksy 5y ago

I have at a few companies now. It's great for processing data that goes beyond a single database call, data formatting and presenting something on a page. If you're chaining multiple microservices together you've made a very sloppy/poor mans version of this. (People tend not to account for downed services, maintence updates, client durability, etc) When you bring in technologies like Kafka to orchestrate this, you'll end up with a more reliable system that you can fix if something goes wrong. This more changes the way you think about data and how you present it. Also, it'll increase your uptime because your service's SLA is isolated from what you're processing. (Your service and the persistent storage that is storing state is what people see.. the data being out of date is something you should account for) Schema changes: Generally you don't have that big of a deal because multiple applications get started at once. You should have system level tests to catch that before you go out. Also, application smoke tests help as well. As long as you picked a durable message queue with a framework that'll crash on error, you can fix that, bring up the fix and continue processing through. Dead letter queues: It's more about how you architect more than anything. This is something you should plan for.

sagichmal 5y ago
> When you bring in technologies like Kafka to orchestrate this, you'll end up with a more reliable system that you can fix if something goes wrong. It depends... If your services are servicing typical user requests, and expect responses in O(<1s), then eventing architectures are exactly the opposite of what you want. Using message brokers (Kafka) as an ersatz network layer is the path to sadness -- you want backpressure, RPC semantics, etc. https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half https://programmingisterrible.com/post/162346490883/how-do-y... If you're doing ETC/event semantics, the game is different. But very few orgs actually work that way.
- monksy 5y ago
  > If your services are servicing typical user requests, and expect responses in O(<1s), then eventing architectures are exactly the opposite of what you want. Using message brokers (Kafka) as an ersatz network layer is the path to sadness -- you want backpressure, RPC semantics, etc. Correct. Most cases you need to see what the current stored state is. Processing can happen in the background and update that state without being requested. I'm assuming that you're architecture your system as having an ingest service that brings everything into the topic, multiple apps that'll refine/enhance, the data, and then an app that sends it out to a more permanent storage. Your service endpoint should look at the premant storage to make a response. (Or read and post out updates via a websocket if you want frequent updates to an existing page) Kafka is not a network technology.

by monksy 5y ago

sagichmal 5y ago

> When you bring in technologies like Kafka to orchestrate this, you'll end up with a more reliable system that you can fix if something goes wrong. It depends... If your services are servicing typical user requests, and expect responses in O(<1s), then eventing architectures are exactly the opposite of what you want. Using message brokers (Kafka) as an ersatz network layer is the path to sadness -- you want backpressure, RPC semantics, etc. https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half https://programmingisterrible.com/post/162346490883/how-do-y... If you're doing ETC/event semantics, the game is different. But very few orgs actually work that way.

monksy 5y ago
> If your services are servicing typical user requests, and expect responses in O(<1s), then eventing architectures are exactly the opposite of what you want. Using message brokers (Kafka) as an ersatz network layer is the path to sadness -- you want backpressure, RPC semantics, etc. Correct. Most cases you need to see what the current stored state is. Processing can happen in the background and update that state without being requested. I'm assuming that you're architecture your system as having an ingest service that brings everything into the topic, multiple apps that'll refine/enhance, the data, and then an app that sends it out to a more permanent storage. Your service endpoint should look at the premant storage to make a response. (Or read and post out updates via a websocket if you want frequent updates to an existing page) Kafka is not a network technology.