3 ms·

If I'm writing all my messages to durable storage why not work off the durable storage? I'm definitely not an expert in this area so perhaps I'm missing someth

by treis 6y ago

If I'm writing all my messages to durable storage why not work off the durable storage? I'm definitely not an expert in this area so perhaps I'm missing something. My logic is that if you're paying the resource cost to write all your messages why not pay the resource cost to read/write back there?

dsies 6y ago
I think you're asking why don't we just be the hosted kafka/rabbitmq/etc and offer all of this stuff in one place. (let me know if that's wrong). That's a totally legit point - we've talked about offering it all in-house before but it would require us to split our efforts into two - operating a PaaS (for a bunch of different messaging tech) and running the event collection platform. Operating the PaaS part would be a full-time effort and there's a lot of competition out there. We've decided to focus on the observability/replay part first (since there is a lot less competition) and then later maybe explore the hosted bus option. LMK if that's not what you meant :)
- treis 6y ago
  >I think you're asking why don't we just be the hosted kafka/rabbitmq/etc and offer all of this stuff in one place. The other way around. If I'm not storing my messages today it's probably because it is too expensive in terms of storage or compute to do so. But, presumably, you can't do that any cheaper than I can. And now we are duplicating the work so even more resources are being consumed making it that much more expensive than just doing it myself. It seems like your service is something I'd want to run pointed towards my Kafka/RabbitMQ/whatever servers. I don't see how duplicating that stream is cost effective.
  - dsies 6y ago
    Ahh gotcha. If you need event introspection, doing it in-house is extremely likely to be more expensive (and definitely time consuming) than offloading it. For example: if you are sending serialized data on your bus - you will need to write something that will deserialize it before inserting it into your elastic search cluster - and now you're managing even more infra (message systems, decoders, document storage). There is definitely a price attached to the luxury - but we're betting that it'll be significantly less than doing it yourself.

by treis 6y ago

dsies 6y ago

I think you're asking why don't we just be the hosted kafka/rabbitmq/etc and offer all of this stuff in one place. (let me know if that's wrong). That's a totally legit point - we've talked about offering it all in-house before but it would require us to split our efforts into two - operating a PaaS (for a bunch of different messaging tech) and running the event collection platform. Operating the PaaS part would be a full-time effort and there's a lot of competition out there. We've decided to focus on the observability/replay part first (since there is a lot less competition) and then later maybe explore the hosted bus option. LMK if that's not what you meant :)

treis 6y ago
>I think you're asking why don't we just be the hosted kafka/rabbitmq/etc and offer all of this stuff in one place. The other way around. If I'm not storing my messages today it's probably because it is too expensive in terms of storage or compute to do so. But, presumably, you can't do that any cheaper than I can. And now we are duplicating the work so even more resources are being consumed making it that much more expensive than just doing it myself. It seems like your service is something I'd want to run pointed towards my Kafka/RabbitMQ/whatever servers. I don't see how duplicating that stream is cost effective.
- dsies 6y ago
  Ahh gotcha. If you need event introspection, doing it in-house is extremely likely to be more expensive (and definitely time consuming) than offloading it. For example: if you are sending serialized data on your bus - you will need to write something that will deserialize it before inserting it into your elastic search cluster - and now you're managing even more infra (message systems, decoders, document storage). There is definitely a price attached to the luxury - but we're betting that it'll be significantly less than doing it yourself.