6 ms·

Show HN: Pyper – Concurrent Python Made Simple

156 points by pyper-dev 2y ago

Hello and happy new year! We're excited to introduce the Pyper package for concurrency & parallelism in Python. Pyper is a flexible framework for concurrent / parallel data processing, following the functional paradigm. Source code can be found on [github](https://github.com/pyper-dev/pyper https://github.com/pyper-dev/pyper) Key features: Intuitive API: Easy to learn, easy to think about. Implements clean abstractions to seamlessly unify threaded, multiprocessed, and asynchronous work. Functional Paradigm: Python functions are the building blocks of data pipelines. Let's you write clean, reusable code naturally. Safety: Hides the heavy lifting of underlying task execution and resource clean-up. No more worrying about race conditions, memory leaks, or thread-level error handling. Efficiency: Designed from the ground up for lazy execution, using queues, workers, and generators. Pure Python: Lightweight, with zero sub-dependencies. We'd love to hear any feedback on this project!

minig33 2y ago
This is cool - I’ve been looking for something like this. I really liked the syntax of Prefect v1 but it was overcomplicated with execution configuration in subsequent versions. I just want something to help me just run async pipelines and prevent AsyncIO weirdness - going to test this out.
grandma_tea 2y ago
Nice! I'm looking forward to trying it out. This seems very similar to https://github.com/cgarciae/pypeln/ https://github.com/cgarciae/pypeln/
- pyper-dev 2y ago
  We came across this at one point and thought it was a very innovative and interesting package! The important design point we're differing on is that Pyper implements 'pipelines' as functions, whereas pypeln seems to implement 'pipelines' as iterable objects.
solidasparagus 2y ago
Nice work! There is a gap when it comes to writing single-machine, concurrent CPU-bound python code. Ray is too big, pykka is threads only, builtins are poorly abstracted. The syntax is also very nice! But I'm not sure I can use this even though I have a specific use-case that feels like it would work well (high-performance pure Python downloading from cloud object storage). The examples are a bit too simple and I don't understand how I can do more complicated things. I chunk up my work, run it in parallel and then I need to do a fan-in step to reduce my chunks - how do you do that in Pyper? Can the processes have state? Pure functions are nice, but if I'm reaching for multiprocess, I need performance and if I need performance, I'll often want a cache of some sort (I don't want to pickle and re-instantiate a cloud client every time I download some bytes for instance). How do exceptions work? Observability? Logs/prints? Then there's stuff that is probably asking too much from this project, but I get it if I write my own python pipeline so it matters to me - rate limiting WIP, cancellation, progress bars. But if some of these problems are/were solved and it offers an easy way to use multiprocessing in python, I would probably use it!
- halfcat 2y ago
  > I don't want to pickle and re-instantiate a cloud client every time I download some bytes for instance Have you tried multiprocessing.shared_memory to address this?
  - solidasparagus 2y ago
    I haven't played with that much! This isn't really a problem in general for my approach to writing this sort of code - when I use multiprocessing, I use a Process class or a worker task function with a setup step followed by a while loop that pulls from a work/control queue. But in the Pyper functional programming world, it would be a concern. IIRC multiprocessing.shared_memory is a much more low-level of abstraction than most python stuff, so I think I'd need to figure out how to make the client use the shared memory and I'm not sure if I could.
- globular-toast 2y ago
  Do you really need to reinvent the wheel every time for parallel workloads? Just learn GNU parallel and write single-threaded code. Concurrency in general isn't about parallelism. It's just about doing multiple things at the same time.
rtpg 2y ago
You really should dive more into the `multiprocess` support option and highlight how this gets around issues with the GIL. This feels like a major value add, and "does this help with CPU-bound work" being "yes" is a big deal! I don't really need pipelining that much, but pipelining along with a certain level of durability and easy multiprocessing support? Now we're talking
- t43562 2y ago
  ...although python 3.13 can be built without the GIL and it really does make threading useful. I did some comparisons with and without. I suppose one excellent thing about this would be if you could just change 1 parameter and switch from multiprocessing to threaded.
  - rtpg 2y ago
    I think you could build off of threading. I do think here it's good to acknowledge that Python async is fundamentally single-threaded (or rather, "single thread per event loop"), so if you do go a multi-thread version you might have to do some bookkeeping to make it all work well. I'm not sure how well async Python libs are tested against working in a world with multiple event loops, but I bet there are a _lot_ of latent bugs in that space.
kissgyorgy 2y ago
Very simple and elegant API!
yablak 2y ago
How does this compare to https://github.com/svenkreiss/pysparkling https://github.com/svenkreiss/pysparkling?
urduntupu 2y ago
Very good README.md, teasing and explaining very well the the provided value. Well done!
giancarlostoro 2y ago
Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)
- morkalork 2y ago
  Wrap the statement in (...) and you can drop the backslashes. See also how people split up complicated loc queries in pandas on multiple lines too.
  - giancarlostoro 2y ago
    Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though). I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.
    - Rickster35 2y ago
      Yeah, this seems neatest if you don't like line breaks pipeline = ( task(get_data, branch=True) | task(step1, workers=20) | task(step2, workers=20) | task(step3, workers=20, multiprocess=True) ) Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.
      - giancarlostoro 2y ago
        Yeah I should have noted that they behave the same in terms of cleaning up a line, not necessarily in terms of what the compiler will do with each token pair.
      - asplake 2y ago
JackC 2y ago
Cool, I was just looking for something like this! It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was: - load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.
- zenapollo 2y ago
  I stumbled on grequests for this use case and it just works.
ge96 2y ago
I still gotta wrap my head around concurrency, I'm not sure if using threads count for concurrency
- gpderetta 2y ago
  of course it does, why wouldn't they?
  - ge96 2y ago
    Something about GIL and how code is written (one statement comes before the other), true parallelism, anyway I gotta do my hw I have not done anything significant like HFT to really dig deep into this Also coming from JS async/await (nodeJS has one thread)
gpderetta 2y ago
From just a short skimming of the docs: - my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio. - I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement. - Async need not force the full pipeline to be async. There should be an option to run async funcitons in background event loops. Especially as you already support threaded executions.
- d0mine 2y ago
  asyncio.TaskGroup from stdlib provides structured concurrency
  - gpderetta 2y ago
    Since 3.11 it seems; I'm currently stuck on 3.8, but hopefully we should be able to upgrade soon. Thanks.
- pyper-dev 2y ago
  Your third point intrigues me a lot. I imagine for the majority of cases, it's generally more useful to work with async functions in the structure of async syntax, but I suppose it's possible to run async functions in a synchronous pipeline. Even though there's currently no built-in support for this, a workaround could be to just define synchronous helper functions to handle running your async logic in an event loop.
deleted 2y ago
[deleted]
jeremieca2 2y ago
[dead]

156 points by pyper-dev 2y ago

minig33 2y ago

This is cool - I’ve been looking for something like this. I really liked the syntax of Prefect v1 but it was overcomplicated with execution configuration in subsequent versions. I just want something to help me just run async pipelines and prevent AsyncIO weirdness - going to test this out.

grandma_tea 2y ago

Nice! I'm looking forward to trying it out. This seems very similar to https://github.com/cgarciae/pypeln/ https://github.com/cgarciae/pypeln/

pyper-dev 2y ago
We came across this at one point and thought it was a very innovative and interesting package! The important design point we're differing on is that Pyper implements 'pipelines' as functions, whereas pypeln seems to implement 'pipelines' as iterable objects.

solidasparagus 2y ago

Nice work! There is a gap when it comes to writing single-machine, concurrent CPU-bound python code. Ray is too big, pykka is threads only, builtins are poorly abstracted. The syntax is also very nice! But I'm not sure I can use this even though I have a specific use-case that feels like it would work well (high-performance pure Python downloading from cloud object storage). The examples are a bit too simple and I don't understand how I can do more complicated things. I chunk up my work, run it in parallel and then I need to do a fan-in step to reduce my chunks - how do you do that in Pyper? Can the processes have state? Pure functions are nice, but if I'm reaching for multiprocess, I need performance and if I need performance, I'll often want a cache of some sort (I don't want to pickle and re-instantiate a cloud client every time I download some bytes for instance). How do exceptions work? Observability? Logs/prints? Then there's stuff that is probably asking too much from this project, but I get it if I write my own python pipeline so it matters to me - rate limiting WIP, cancellation, progress bars. But if some of these problems are/were solved and it offers an easy way to use multiprocessing in python, I would probably use it!

halfcat 2y ago
> I don't want to pickle and re-instantiate a cloud client every time I download some bytes for instance Have you tried multiprocessing.shared_memory to address this?
- solidasparagus 2y ago
  I haven't played with that much! This isn't really a problem in general for my approach to writing this sort of code - when I use multiprocessing, I use a Process class or a worker task function with a setup step followed by a while loop that pulls from a work/control queue. But in the Pyper functional programming world, it would be a concern. IIRC multiprocessing.shared_memory is a much more low-level of abstraction than most python stuff, so I think I'd need to figure out how to make the client use the shared memory and I'm not sure if I could.
globular-toast 2y ago
Do you really need to reinvent the wheel every time for parallel workloads? Just learn GNU parallel and write single-threaded code. Concurrency in general isn't about parallelism. It's just about doing multiple things at the same time.

rtpg 2y ago

You really should dive more into the `multiprocess` support option and highlight how this gets around issues with the GIL. This feels like a major value add, and "does this help with CPU-bound work" being "yes" is a big deal! I don't really need pipelining that much, but pipelining along with a certain level of durability and easy multiprocessing support? Now we're talking

t43562 2y ago
...although python 3.13 can be built without the GIL and it really does make threading useful. I did some comparisons with and without. I suppose one excellent thing about this would be if you could just change 1 parameter and switch from multiprocessing to threaded.
- rtpg 2y ago
  I think you could build off of threading. I do think here it's good to acknowledge that Python async is fundamentally single-threaded (or rather, "single thread per event loop"), so if you do go a multi-thread version you might have to do some bookkeeping to make it all work well. I'm not sure how well async Python libs are tested against working in a world with multiple event loops, but I bet there are a _lot_ of latent bugs in that space.

kissgyorgy 2y ago

Very simple and elegant API!

yablak 2y ago

How does this compare to https://github.com/svenkreiss/pysparkling https://github.com/svenkreiss/pysparkling?

urduntupu 2y ago

Very good README.md, teasing and explaining very well the the provided value. Well done!

giancarlostoro 2y ago

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)

morkalork 2y ago
Wrap the statement in (...) and you can drop the backslashes. See also how people split up complicated loc queries in pandas on multiple lines too.
- giancarlostoro 2y ago
  Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though). I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.
  - Rickster35 2y ago
    Yeah, this seems neatest if you don't like line breaks pipeline = ( task(get_data, branch=True) | task(step1, workers=20) | task(step2, workers=20) | task(step3, workers=20, multiprocess=True) ) Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.
    - giancarlostoro 2y ago
      Yeah I should have noted that they behave the same in terms of cleaning up a line, not necessarily in terms of what the compiler will do with each token pair.
    - asplake 2y ago

JackC 2y ago

Cool, I was just looking for something like this! It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was: - load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.

zenapollo 2y ago
I stumbled on grequests for this use case and it just works.

ge96 2y ago

I still gotta wrap my head around concurrency, I'm not sure if using threads count for concurrency

gpderetta 2y ago
of course it does, why wouldn't they?
- ge96 2y ago
  Something about GIL and how code is written (one statement comes before the other), true parallelism, anyway I gotta do my hw I have not done anything significant like HFT to really dig deep into this Also coming from JS async/await (nodeJS has one thread)

gpderetta 2y ago

From just a short skimming of the docs: - my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio. - I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement. - Async need not force the full pipeline to be async. There should be an option to run async funcitons in background event loops. Especially as you already support threaded executions.

d0mine 2y ago
asyncio.TaskGroup from stdlib provides structured concurrency
- gpderetta 2y ago
  Since 3.11 it seems; I'm currently stuck on 3.8, but hopefully we should be able to upgrade soon. Thanks.
pyper-dev 2y ago
Your third point intrigues me a lot. I imagine for the majority of cases, it's generally more useful to work with async functions in the structure of async syntax, but I suppose it's possible to run async functions in a synchronous pipeline. Even though there's currently no built-in support for this, a workaround could be to just define synchronous helper functions to handle running your async logic in an event loop.

deleted 2y ago

[deleted]

jeremieca2 2y ago

[dead]