3 ms·
Python actually has great support for pipelining with the lazy iteration protocol built into the language. The problem arises when you need to chain executable
by nerdponx 5y ago
Python actually has great support for pipelining with the lazy iteration protocol built into the language.
The problem arises when you need to chain executables together, rather than just Python code. It turns out that shells do some pretty complicated stuff with file descriptors and forking, stuff that you definitely do not want to DIY in Python. So "streaming" data from one executable to another means you buffer everything in Python runtime, even if you use async. This is fine for a lot of cases, but it feels wasteful and in elegant to me.