4 ms·

Show HN: Maestro – an Erlang pool of pools

26 points by g-andrade 11y ago

felixgallo 11y ago
It's an interesting concept. Did you try, e.g., dispcount (https://github.com/ferd/dispcount https://github.com/ferd/dispcount) and find it not up to par for your use case?
- g-andrade 11y ago
  Read this in the description: "If you cannot afford to ignore a query and wish to eventually serve every one of them, dispcount might not be for you." Which does not fit my current problems; in any case, it seems to be a very promising project, I had no knowledge of it. Good suggestion.
SingAlong 11y ago
Found this in the README: "... pick them at random when checking out ..." Isn't a Round-Robin-based checkout better than randomly picking pools to checkout from?
- jerf 11y ago
  To do round-robin correctly would re-introduce the problem this is created to solve in the first place, which is that you can't afford to have a single-process bottleneck around your pool. (That is to say, we are in a situation where that has proved to be a problem. Normally worrying about this would constitute "premature optimization" if you haven't yet proved it to be a problem in your current program.) To do it without a lock becomes theoretically indistinguishable from "random", in a sense, but a pretty real sense at scale. This is, IMHO, one of the weaknesses in the Erlang shared-nothing process model. At that layer of abstraction, you can't directly implement an efficient multithreading-aware pool. A primitive that would allow some sort of pool-like resource acquisition to efficiently claim and put back a resource, implemented in the VM where it can use the full power of the processor, would be very useful in a number of situations. (There's a number of paths to such a thing; implementing a shared mailbox that many processes could listen on with guaranteed-once delivery (obviously this can only be done with local mailboxes), the obvious implementation of something that is simply a pool, perhaps others that are more clever or simple or enable other interesting cases.) I've encountered similar problems myself. (To be clear, in the grand scheme of things, this is a relatively minor issue, but, as you scale up, is one that you do have a certain likelihood of encountering.)
- g-andrade 11y ago
  What Jerf said. :)
g-andrade 11y ago
Creator here; it's worth pointing out that the current picking solution might suffer from its own (performance, not scalability) problems - it reads the pools info from a shared ETS table and picks a random element from that list; mandatory, deterministic naming of the pools would limit this step to a simple concatenation / conversion to atom. Something I will consider.

26 points by g-andrade 11y ago

It's an interesting concept. Did you try, e.g., dispcount (https://github.com/ferd/dispcount https://github.com/ferd/dispcount) and find it not up to par for your use case?

g-andrade 11y ago
Read this in the description: "If you cannot afford to ignore a query and wish to eventually serve every one of them, dispcount might not be for you." Which does not fit my current problems; in any case, it seems to be a very promising project, I had no knowledge of it. Good suggestion.

SingAlong 11y ago

Found this in the README: "... pick them at random when checking out ..." Isn't a Round-Robin-based checkout better than randomly picking pools to checkout from?

jerf 11y ago
To do round-robin correctly would re-introduce the problem this is created to solve in the first place, which is that you can't afford to have a single-process bottleneck around your pool. (That is to say, we are in a situation where that has proved to be a problem. Normally worrying about this would constitute "premature optimization" if you haven't yet proved it to be a problem in your current program.) To do it without a lock becomes theoretically indistinguishable from "random", in a sense, but a pretty real sense at scale. This is, IMHO, one of the weaknesses in the Erlang shared-nothing process model. At that layer of abstraction, you can't directly implement an efficient multithreading-aware pool. A primitive that would allow some sort of pool-like resource acquisition to efficiently claim and put back a resource, implemented in the VM where it can use the full power of the processor, would be very useful in a number of situations. (There's a number of paths to such a thing; implementing a shared mailbox that many processes could listen on with guaranteed-once delivery (obviously this can only be done with local mailboxes), the obvious implementation of something that is simply a pool, perhaps others that are more clever or simple or enable other interesting cases.) I've encountered similar problems myself. (To be clear, in the grand scheme of things, this is a relatively minor issue, but, as you scale up, is one that you do have a certain likelihood of encountering.)
g-andrade 11y ago
What Jerf said. :)

g-andrade 11y ago

Creator here; it's worth pointing out that the current picking solution might suffer from its own (performance, not scalability) problems - it reads the pools info from a shared ETS table and picks a random element from that list; mandatory, deterministic naming of the pools would limit this step to a simple concatenation / conversion to atom. Something I will consider.