7 ms·

I switched from web dev to data science some years ago, and surprisingly couldn't find a streaming parallelizer for Python -- every package assumes you loaded t

I switched from web dev to data science some years ago, and surprisingly couldn't find a streaming parallelizer for Python -- every package assumes you loaded the whole dataset in memory. Had to write my own.

elashri 3y ago
There are many packages that can do that, like Vax [1] and Dask [2]. I don't know exactly your workflow. But the concurrency in python is limited to multiprocessing, which is much expensive than threads which usually a typical streaming parallelizer will use outside python world. [1] https://vaex.io/docs/index.html https://vaex.io/docs/index.html [2] https://docs.dask.org/en/latest/ https://docs.dask.org/en/latest/
bboygravity 3y ago
Just curious: why would you not load the entire dataset "into memory" ("into memory" from a Python perspective)? On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. Or are the datasets large enough to fit on neither?
- ninkendo 3y ago
  > On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. This is not true, unless you’re referring to swap (which is a configuration of the system and may not be big enough to actually fit it either, many people run with only a small amount of swap or disable it altogether.) You may be referring to mmap(2), which will map the on-disk dataset to a region of memory that is paged in on-demand, but somehow I doubt that’s what OP was referring to either. If you just read() the file into memory and work on it, you’re going to be using a ton of RAM. The OS will only put “the rest on disk” if it swaps, which is a degenerate performance case, and it may not even be the dataset itself that gets swapped (the kernel may opt to swap everything else on the system to fit your dataset into RAM. All pages are equal in the eyes of the virtual memory layer, and the ranking algorithm is basically an LRU cache.)
  - geocar 3y ago
    read() doesn't read from the disk if the blocks are already in memory.
    - ninkendo 3y ago
      That’s a really misleading thing to say. If the kernel already has the thing you’re read()’ing cached, then yes the kernel can skip the disk read as an optimization. But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space, which makes it no longer just a “cache”. You’re now “using memory”. read() is not mmap(). You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
      - geocar 3y ago
        > If the kernel already has the thing you’re read()’ing cached which it would do, if you have just downloaded the file. > But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. > You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way. i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
        ninkendo 3y ago
        Please keep in mind the context of the discussion. A prior poster made a claim that they can read a file into memory, and that it won’t actually use any additional memory because the kernel will “automatically take care” of it somehow. This is plainly false. You come in and say something to the effect of “but it may not have to read from disk because it’s cached”, which… has nothing to do with what was being discussed. We’re not talking about whether it incurs a disk read, we’re talking about whether it will run your system out of memory trying to load it into RAM. > i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. What in god’s holy name are you blathering about? If I “just downloaded it from the network”, it’s on-disk. If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk. If I read() the contents, which is what you said I should do, then another copy of the data is now sitting in a buffer in my process’s heap. This extra copy is now "using" memory, and if I keep doing this, I will run the system out of RAM. This is characteristically different from mmap(), where a region of memory maps to a file on-disk, and contents are faulted into memory as I read them. The reason this is an extremely important distinction, is that in the mmap scenario, the kernel is free to free the read-in pages any time it wants, and they will be faulted back again in if I try to read them again. Contrast this with using read(), which makes it so the kernel can't free the pages, because they're buffers in my process's heap, and are not considered file-backed from the kernel's perspective. > i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also. Swap is disabled on my system. Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it (because the data is in my heap, and thus isn’t file-backed, so the kernel can’t just free the pages and read them back from the file I read it from.) read() is not mmap(). Please don’t conflate them.
- fidotron 3y ago
  This is actually the old SAX vs DOM xml parsing discussion in disguise. SAX is harder but has at least two key strongly related benefits: 1. Can handle a continuous firehose 2. Processing can start before the load is completed (because it might never be completed) so the time to first useful action can be greatly reduced.
- thedougd 3y ago
  Example: I'm querying a database, producing a file and storing it in object storage (S3). The dataset is 100 gigabytes in size. I should not require 100 gigabytes of memory or disk space to handle this single operation. It would be slower to write it to disk first.
- tsimionescu 3y ago
  Depending on the source of the data, that is not as good as an actual streaming implementation. That is, if the data is coming from a network API, waiting for it to be "in memory" before processing it still means that you have to store the whole stream on the local machine before you even start. Even if we assume that you are storing it on disk and mmapping it into memory that's still not a good idea for many use cases. Not to mention, if the code is not explicitly designed to work with a streaming approach, even for local data, might mean that early steps accidentally end up touching the whole data (e.g. they look for a closing } in something like a 10GB json document) in unexpected places, costing orders of magnitude more than they should.
  - Thiez 3y ago
    JSON discourages but does not forbid duplicate keys. In case of duplicate keys browsers generally let the last instance win. So if you want to be compatible with that, you always must read the whole document.
- culebron21 3y ago
  I had 5..10GB geopackage, and either hadn't spare 10..20GB of RAM, or had to make sure my data processing routine would fut in a busy server.
otteromkram 3y ago
Did you try dask? https://www.dask.org/ https://www.dask.org/
- deleted 3y ago
  [deleted]
- culebron21 3y ago
  I've looked into the samples and recall what the problem was: geopandas was in an experimental branch, and you had to lock yourself into dask -- plus, the geopandas code had to be rewritten completely for dask. So i wrote my own processor that applies the same function in map&reduce fashion, and keeps code compatible with jupyter notebooks -- you decorate functions to be parallelizeable, but still import them and call normally. https://github.com/culebron/erde https://github.com/culebron/erde
goeiedaggoeie 3y ago
Same in video parsers and tooling frequently, expects a whole mp4 to be there, or a whole video to parse it, yet gstreamer/ffmpegapi delivers the content as a stream of buffers that you have to process one buffer at a time.
- prox 3y ago
  Is it me or aren’t there a whole lot video specialists in general? It’s just something I noticed here and there on github.
  - londons_explore 3y ago
    In a way, that's good. The few hundred video encoding specialists who exist in the world have, per person, had a huge impact on the world. Compare that to web developers, who in total have had probably a larger impact on the world, but per head it is far lower. Part of engineering is to use the fewest people possible to have the biggest benefit for the most people. Video did that well - I suspect partly by being 'hard'.
  - nullpilot 3y ago
    My experience that played out over the last few weeks lead me to a similar belief, somewhat. For rather uninteresting reasons I decided I wanted to create mp4 videos of an animation programmatically. The first solution suggested when googling around is to just create all the frames, save them to disk, and then let ffmpeg do its thing from there. I would have just gone with that for a one-off task, but it's a pretty bad solution if the video is long, or high res, or both. Plus, what I really wanted was to build something more "scalable/flexible". Maybe I didn't know the right keywords to search for, but there really didn't seem to be many options for creating frames, piping them straight to an encoder, and writing just the final video file to disk. The only one I found that seemed like it could maybe do it the way I had in mind was VidGear[1] (Python). I had figured that with the popularity of streaming, and video in general on the web, there would be so much more tooling for these sorts of things. I ended up digging way deeper into this than I had intended, and built myself something on top of Membrane[2] (Elixir) [1] https://abhitronix.github.io/vidgear/ https://abhitronix.github.io/vidgear/ [2] https://membrane.stream/ https://membrane.stream/

elashri 3y ago

There are many packages that can do that, like Vax [1] and Dask [2]. I don't know exactly your workflow. But the concurrency in python is limited to multiprocessing, which is much expensive than threads which usually a typical streaming parallelizer will use outside python world. [1] https://vaex.io/docs/index.html https://vaex.io/docs/index.html [2] https://docs.dask.org/en/latest/ https://docs.dask.org/en/latest/

bboygravity 3y ago

Just curious: why would you not load the entire dataset "into memory" ("into memory" from a Python perspective)? On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. Or are the datasets large enough to fit on neither?

ninkendo 3y ago
> On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. This is not true, unless you’re referring to swap (which is a configuration of the system and may not be big enough to actually fit it either, many people run with only a small amount of swap or disable it altogether.) You may be referring to mmap(2), which will map the on-disk dataset to a region of memory that is paged in on-demand, but somehow I doubt that’s what OP was referring to either. If you just read() the file into memory and work on it, you’re going to be using a ton of RAM. The OS will only put “the rest on disk” if it swaps, which is a degenerate performance case, and it may not even be the dataset itself that gets swapped (the kernel may opt to swap everything else on the system to fit your dataset into RAM. All pages are equal in the eyes of the virtual memory layer, and the ranking algorithm is basically an LRU cache.)
- geocar 3y ago
  read() doesn't read from the disk if the blocks are already in memory.
  - ninkendo 3y ago
    That’s a really misleading thing to say. If the kernel already has the thing you’re read()’ing cached, then yes the kernel can skip the disk read as an optimization. But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space, which makes it no longer just a “cache”. You’re now “using memory”. read() is not mmap(). You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
    - geocar 3y ago
      > If the kernel already has the thing you’re read()’ing cached which it would do, if you have just downloaded the file. > But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. > You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way. i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
      - ninkendo 3y ago
        Please keep in mind the context of the discussion. A prior poster made a claim that they can read a file into memory, and that it won’t actually use any additional memory because the kernel will “automatically take care” of it somehow. This is plainly false. You come in and say something to the effect of “but it may not have to read from disk because it’s cached”, which… has nothing to do with what was being discussed. We’re not talking about whether it incurs a disk read, we’re talking about whether it will run your system out of memory trying to load it into RAM. > i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. What in god’s holy name are you blathering about? If I “just downloaded it from the network”, it’s on-disk. If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk. If I read() the contents, which is what you said I should do, then another copy of the data is now sitting in a buffer in my process’s heap. This extra copy is now "using" memory, and if I keep doing this, I will run the system out of RAM. This is characteristically different from mmap(), where a region of memory maps to a file on-disk, and contents are faulted into memory as I read them. The reason this is an extremely important distinction, is that in the mmap scenario, the kernel is free to free the read-in pages any time it wants, and they will be faulted back again in if I try to read them again. Contrast this with using read(), which makes it so the kernel can't free the pages, because they're buffers in my process's heap, and are not considered file-backed from the kernel's perspective. > i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also. Swap is disabled on my system. Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it (because the data is in my heap, and thus isn’t file-backed, so the kernel can’t just free the pages and read them back from the file I read it from.) read() is not mmap(). Please don’t conflate them.
fidotron 3y ago
This is actually the old SAX vs DOM xml parsing discussion in disguise. SAX is harder but has at least two key strongly related benefits: 1. Can handle a continuous firehose 2. Processing can start before the load is completed (because it might never be completed) so the time to first useful action can be greatly reduced.
thedougd 3y ago
Example: I'm querying a database, producing a file and storing it in object storage (S3). The dataset is 100 gigabytes in size. I should not require 100 gigabytes of memory or disk space to handle this single operation. It would be slower to write it to disk first.
tsimionescu 3y ago
Depending on the source of the data, that is not as good as an actual streaming implementation. That is, if the data is coming from a network API, waiting for it to be "in memory" before processing it still means that you have to store the whole stream on the local machine before you even start. Even if we assume that you are storing it on disk and mmapping it into memory that's still not a good idea for many use cases. Not to mention, if the code is not explicitly designed to work with a streaming approach, even for local data, might mean that early steps accidentally end up touching the whole data (e.g. they look for a closing } in something like a 10GB json document) in unexpected places, costing orders of magnitude more than they should.
- Thiez 3y ago
  JSON discourages but does not forbid duplicate keys. In case of duplicate keys browsers generally let the last instance win. So if you want to be compatible with that, you always must read the whole document.
culebron21 3y ago
I had 5..10GB geopackage, and either hadn't spare 10..20GB of RAM, or had to make sure my data processing routine would fut in a busy server.

otteromkram 3y ago

Did you try dask? https://www.dask.org/ https://www.dask.org/

deleted 3y ago
[deleted]
culebron21 3y ago
I've looked into the samples and recall what the problem was: geopandas was in an experimental branch, and you had to lock yourself into dask -- plus, the geopandas code had to be rewritten completely for dask. So i wrote my own processor that applies the same function in map&reduce fashion, and keeps code compatible with jupyter notebooks -- you decorate functions to be parallelizeable, but still import them and call normally. https://github.com/culebron/erde https://github.com/culebron/erde

goeiedaggoeie 3y ago

Same in video parsers and tooling frequently, expects a whole mp4 to be there, or a whole video to parse it, yet gstreamer/ffmpegapi delivers the content as a stream of buffers that you have to process one buffer at a time.

prox 3y ago
Is it me or aren’t there a whole lot video specialists in general? It’s just something I noticed here and there on github.
- londons_explore 3y ago
  In a way, that's good. The few hundred video encoding specialists who exist in the world have, per person, had a huge impact on the world. Compare that to web developers, who in total have had probably a larger impact on the world, but per head it is far lower. Part of engineering is to use the fewest people possible to have the biggest benefit for the most people. Video did that well - I suspect partly by being 'hard'.
- nullpilot 3y ago
  My experience that played out over the last few weeks lead me to a similar belief, somewhat. For rather uninteresting reasons I decided I wanted to create mp4 videos of an animation programmatically. The first solution suggested when googling around is to just create all the frames, save them to disk, and then let ffmpeg do its thing from there. I would have just gone with that for a one-off task, but it's a pretty bad solution if the video is long, or high res, or both. Plus, what I really wanted was to build something more "scalable/flexible". Maybe I didn't know the right keywords to search for, but there really didn't seem to be many options for creating frames, piping them straight to an encoder, and writing just the final video file to disk. The only one I found that seemed like it could maybe do it the way I had in mind was VidGear[1] (Python). I had figured that with the popularity of streaming, and video in general on the web, there would be so much more tooling for these sorts of things. I ended up digging way deeper into this than I had intended, and built myself something on top of Membrane[2] (Elixir) [1] https://abhitronix.github.io/vidgear/ https://abhitronix.github.io/vidgear/ [2] https://membrane.stream/ https://membrane.stream/