3 ms·

I apologize for this stupid question but whenever i see products like this or kafka, i cant help but wonder. when exactly do you need a system like this compare

I apologize for this stupid question but whenever i see products like this or kafka, i cant help but wonder. when exactly do you need a system like this compared to a traditional redis pub sub?

esafak 1y ago
What about the transformation in the middle, like a moving average?
acjohnson55 1y ago
It's very useful any time the input to some system is a stream of events, potentially from a whole bunch of different sources, but you want the output to be a unified relational data model. I used to work in insurance, and we had a whole bunch of systems of record for different functions of the business -- CRM, policy management, billing, claims, etc. Some were our own tech, many were SaaS. It's great to be able to keep these systems decoupled operationally. That way, you can replace pieces and have your business areas have fairly independent IT stacks. But many backoffice tasks, like finance, accounting, and servicing need a holistic view of what's going on. It's helpful to ingest all the data into a centralized warehouse, and build up a unified model of the state of the business. A lot of analysts like to write these data transformations in SQL. Insurance is not a fast-paced business, so we largely ingested the data in structured form. But you can imagine that for faster businesses, like advertising, monitoring, IoT, or trading, the data from the systems of record might be an event stream, rather than a data model. These stream processing databases are designed for this type of situation, where you may want real-time ETL, event-by-event. EDIT: Also, their website has a use cases section: https://risingwave.com/use-cases/ https://risingwave.com/use-cases/
redman25 1y ago
Solutions like this can help with complex transformations that rely on intermediate state like streaming aggregations. They often have watermarking, windowing, and all that good stuff built in whereas with redis you would have to build all of that in your application.

I apologize for this stupid question but whenever i see products like this or kafka, i cant help but wonder. when exactly do you need a system like this compared to a traditional redis pub sub?

esafak 1y ago

What about the transformation in the middle, like a moving average?

acjohnson55 1y ago

It's very useful any time the input to some system is a stream of events, potentially from a whole bunch of different sources, but you want the output to be a unified relational data model. I used to work in insurance, and we had a whole bunch of systems of record for different functions of the business -- CRM, policy management, billing, claims, etc. Some were our own tech, many were SaaS. It's great to be able to keep these systems decoupled operationally. That way, you can replace pieces and have your business areas have fairly independent IT stacks. But many backoffice tasks, like finance, accounting, and servicing need a holistic view of what's going on. It's helpful to ingest all the data into a centralized warehouse, and build up a unified model of the state of the business. A lot of analysts like to write these data transformations in SQL. Insurance is not a fast-paced business, so we largely ingested the data in structured form. But you can imagine that for faster businesses, like advertising, monitoring, IoT, or trading, the data from the systems of record might be an event stream, rather than a data model. These stream processing databases are designed for this type of situation, where you may want real-time ETL, event-by-event. EDIT: Also, their website has a use cases section: https://risingwave.com/use-cases/ https://risingwave.com/use-cases/

redman25 1y ago

Solutions like this can help with complex transformations that rely on intermediate state like streaming aggregations. They often have watermarking, windowing, and all that good stuff built in whereas with redis you would have to build all of that in your application.