5 ms·

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

by mej10 10y ago

kchoudhu 10y ago
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...
- vegabook 10y ago
  indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.
  - kchoudhu 10y ago
    For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
    - vegabook 10y ago
      that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the world "samples". Not every single data point is needed though it's nice to have 99.999.. and we usually get that). It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.
      - janczukt 10y ago
        This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.
- oldmanjay 10y ago
  What if they crash without registering an exit?
  - kchoudhu 10y ago
    Python context managers and heartbeats :)
  - vegabook 10y ago
    They broadcast every 2 seconds. No heartbeat = dead.
    - oldmanjay 10y ago
      So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.
      - vegabook 10y ago
        The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all the IPs. Then in the main loop you just make sure that you've gotten a recent pulse otherwise you reconnect. If you're using zmq it's just another short poller entry amongst the others that you will already inevitably have. Literally a couple of lines. It allows you to bring nodes up and down at will for robustness and scalability. I think it's much less complex than running an entire database node just for this, which btw will also require you constantly to poll, and will require you to bring in an (often heavyweight) client library into each node too, as opposed to standard-library sockets which if you're running multinode you're almost certainly already importing. If you're looking for "simple" distributed computing my sense is that that has yet to be invented.
vegabook 10y ago
That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

by mej10 10y ago

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...

vegabook 10y ago
indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.
- kchoudhu 10y ago
  For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
  - vegabook 10y ago
    that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the world "samples". Not every single data point is needed though it's nice to have 99.999.. and we usually get that). It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.
    - janczukt 10y ago
      This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.
oldmanjay 10y ago
What if they crash without registering an exit?
- kchoudhu 10y ago
  Python context managers and heartbeats :)
- vegabook 10y ago
  They broadcast every 2 seconds. No heartbeat = dead.
  - oldmanjay 10y ago
    So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.
    - vegabook 10y ago
      The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all the IPs. Then in the main loop you just make sure that you've gotten a recent pulse otherwise you reconnect. If you're using zmq it's just another short poller entry amongst the others that you will already inevitably have. Literally a couple of lines. It allows you to bring nodes up and down at will for robustness and scalability. I think it's much less complex than running an entire database node just for this, which btw will also require you constantly to poll, and will require you to bring in an (often heavyweight) client library into each node too, as opposed to standard-library sockets which if you're running multinode you're almost certainly already importing. If you're looking for "simple" distributed computing my sense is that that has yet to be invented.

vegabook 10y ago

That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.