4 ms·

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way

by devxpy 8y ago

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/ https://github.com/pycampers/zproc/

heavenlyblue 8y ago
>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?
- TickleSteve 8y ago
  No, he's reinvented multiprocessing... pickling data structures across multiple processes. Just without the 'niceties'.
  - zbentley 8y ago
    You're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.
    - TickleSteve 8y ago
      I agree, multiprocessing will be more performant than ZProc, much more thought has gone into it than the simple 0mq wrapper that is ZProc.
      - devxpy 8y ago
        Exactly.
- devxpy 8y ago
  Great, now just "stating" how things work is equivalent to inventing them!
TickleSteve 8y ago
You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. (https://en.wikipedia.org/wiki/Message_passing https://en.wikipedia.org/wiki/Message_passing) In other words... its not a revolution. ZProc seems to simply be a simple library to pickle data structures thru a central (pubsub?) server. This is not the way to get remotely close to "high performance". What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved).
- zbentley 8y ago
  > What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where you have all your data before creating your processes/pool and just want to farm it out to your MP processes without waiting for it all to be chunked/pickled/unpickled. Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. This pattern can be used to totally bypass all considerations of performance/CPU/etc. for pickling/unpickling data and lends a massive speed boost in certain situations--e.g. a massive dataset is read into memory at startup, and then ranges of that dataset are processed in parallel by a pool of MP processes, each of which will return a relatively small result-set back to the parent, or each of which will write its processed (think: data scrubbing) range to a separate file which could be `cat`ed together, or written in parallel with careful `seek` bookkeeping. Unix-ish OSes only, though (unless the fork() emulation in WSL works for this--I have not tested that). * Technically it's O(N) for the size of data you have in memory at process pool start, because fork() can take time, but the multiplier is small enough in practice compared to sending data to/from MP processes via queues or whatever that it might as well be constant.
adw 8y ago
> My library lets you do parallelism in a unique way That's a big claim which you don't really back up as much as you need to. Unique is an extremely high bar in this very busy field. There are several other similar red flags on the linked GitHub; I think your enthusiasm is running away from you a little. You might want to dial the ten-dollar language back a bit – it made me immediately suspicious ("utterly perfect", for example is another danger phrase). It's the combination of grandiose language + solution-in-search-of-a-problem which leads to that. If you're going to sell hard, what I would want to see is a large, complex, high-traffic system which makes extensive use of this; if you compare and contrast with Ray, which I've also only just encountered in this thread, there's a real problem (distributed hyperparameter optimization) which they've built a solution for with the library, and that immediately lends it credibility; I know the system can be used for something because it has been.
- devxpy 8y ago
  'utterly perfect ' are not my words http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Thought linking it there would make it better, but I'll just remove it... And you do make a good point. It doesn't really solve anything technically. But would you agree that it exposes a better API for doing much of the same stuff?
  - adw 8y ago
    I wouldn't know without using it. That's where "software using this library" is a really useful bit of social proof. Think of Django; even without looking at the code you have a lot of evidence that it can conveniently solve a wide range of real problems.
    - devxpy 8y ago
      Well HN I'd say is a pretty good place to raise social awareness
- devxpy 8y ago
  Update - I hopefully made the language a little better? https://github.com/pycampers/zproc/blob/master/README.md https://github.com/pycampers/zproc/blob/master/README.md

by devxpy 8y ago

heavenlyblue 8y ago

>> Zproc uses a Server, which is responsible for storing and communicating the state. >> >> This isolates our resource (state), eliminating the need for locks. So you've just invented a new name for a coordinator process and called it a new fashion in computation?

TickleSteve 8y ago
No, he's reinvented multiprocessing... pickling data structures across multiple processes. Just without the 'niceties'.
- zbentley 8y ago
  You're probably right, but see my comment above: not only is MP possibly superior at being a picking/arbitrating server, but it also supports taking advantage of copy-on-write semantics on Unix-ish systems to transfer memory to children at startup in constant time with no pickling/unpickling necessary.
  - TickleSteve 8y ago
    I agree, multiprocessing will be more performant than ZProc, much more thought has gone into it than the simple 0mq wrapper that is ZProc.
    - devxpy 8y ago
      Exactly.
devxpy 8y ago
Great, now just "stating" how things work is equivalent to inventing them!

TickleSteve 8y ago

You make some extremely large claims about ZProc, what advantages does it have over every other message-passing library for every other language ever built? (including the other zeromq bindings?) TBH, you're claims sound like you've just "discovered" message-passing, of which many, many languages, runtimes and operating systems have been using for many years/decades. (https://en.wikipedia.org/wiki/Message_passing https://en.wikipedia.org/wiki/Message_passing) In other words... its not a revolution. ZProc seems to simply be a simple library to pickle data structures thru a central (pubsub?) server. This is not the way to get remotely close to "high performance". What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved).

zbentley 8y ago
> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where you have all your data before creating your processes/pool and just want to farm it out to your MP processes without waiting for it all to be chunked/pickled/unpickled. Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. This pattern can be used to totally bypass all considerations of performance/CPU/etc. for pickling/unpickling data and lends a massive speed boost in certain situations--e.g. a massive dataset is read into memory at startup, and then ranges of that dataset are processed in parallel by a pool of MP processes, each of which will return a relatively small result-set back to the parent, or each of which will write its processed (think: data scrubbing) range to a separate file which could be `cat`ed together, or written in parallel with careful `seek` bookkeeping. Unix-ish OSes only, though (unless the fork() emulation in WSL works for this--I have not tested that). * Technically it's O(N) for the size of data you have in memory at process pool start, because fork() can take time, but the multiplier is small enough in practice compared to sending data to/from MP processes via queues or whatever that it might as well be constant.

adw 8y ago

> My library lets you do parallelism in a unique way That's a big claim which you don't really back up as much as you need to. Unique is an extremely high bar in this very busy field. There are several other similar red flags on the linked GitHub; I think your enthusiasm is running away from you a little. You might want to dial the ten-dollar language back a bit – it made me immediately suspicious ("utterly perfect", for example is another danger phrase). It's the combination of grandiose language + solution-in-search-of-a-problem which leads to that. If you're going to sell hard, what I would want to see is a large, complex, high-traffic system which makes extensive use of this; if you compare and contrast with Ray, which I've also only just encountered in this thread, there's a real problem (distributed hyperparameter optimization) which they've built a solution for with the library, and that immediately lends it credibility; I know the system can be used for something because it has been.

devxpy 8y ago
'utterly perfect ' are not my words http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Thought linking it there would make it better, but I'll just remove it... And you do make a good point. It doesn't really solve anything technically. But would you agree that it exposes a better API for doing much of the same stuff?
- adw 8y ago
  I wouldn't know without using it. That's where "software using this library" is a really useful bit of social proof. Think of Django; even without looking at the code you have a lot of evidence that it can conveniently solve a wide range of real problems.
  - devxpy 8y ago
    Well HN I'd say is a pretty good place to raise social awareness
devxpy 8y ago
Update - I hopefully made the language a little better? https://github.com/pycampers/zproc/blob/master/README.md https://github.com/pycampers/zproc/blob/master/README.md