3 ms·

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is complet

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.

kchoudhu 10y ago
For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
- vegabook 10y ago
  that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the world "samples". Not every single data point is needed though it's nice to have 99.999.. and we usually get that). It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.
  - janczukt 10y ago
    This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.

3 ms·

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is complet

by vegabook 10y ago

kchoudhu 10y ago
For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
- vegabook 10y ago
  that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the world "samples". Not every single data point is needed though it's nice to have 99.999.. and we usually get that). It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.
  - janczukt 10y ago
    This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.