4 ms·

Good content, but over SEO optimized. Would be nice to hear about the actual efficiency of using these methods. For instance, does fork() copy the page of memo

by arjvik 3y ago

Good content, but over SEO optimized. Would be nice to hear about the actual efficiency of using these methods. For instance, does fork() copy the page of memory containing the array? I believe it's Copy-on-Write semantics, right? What happens when the parent process changes the array? Then, how do Pipe and Queue send the array across processes? Do they also pickle and unpickle it? Use shared memory?

ikhatri 3y ago
This is a super common occurrence in training loops for ML models because PyTorch uses multiprocessing for its dataloader workers. If you want to read more, see the discussion in this issue: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 https://github.com/pytorch/pytorch/issues/13246#issuecomment... As you’ve pointed out fork() isn’t ideal for a number of reasons and in general it’s preferred to use torch tensors directly instead of numpy arrays so that you are not forced into using fork() There’s also this write up which I found to be quite useful for details: https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/ https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multip...
ahoka 3y ago
“I believe it's Copy-on-Write semantics, right? What happens when the parent process changes the array?” Yes and you just answered your question.
- boppo1 3y ago
  For a programming novice: Is CoW where a 'child' process operates on a copy of data stored elsewhere in memory, but you have some mechanism to always update the copy any time the parent process modifies/writes to the original data?
  - porridgeandrice 3y ago
    No, if I have stuff, and I fork(), you, my child process is expected to have a copy of that stuff in it's current state. However, the kernel lies to you. It delays copying the stuff until I write to it i.e until I need to see a new state. Hence, "copy-on-write". In this case, I am the parent python process, you are the child processes started with the `multiprocessing` module, and the data is the numpy array. The implication is that the changes made by the parent process to the numpy array post fork() won't be visible to the child processes.

by arjvik 3y ago

ikhatri 3y ago

This is a super common occurrence in training loops for ML models because PyTorch uses multiprocessing for its dataloader workers. If you want to read more, see the discussion in this issue: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 https://github.com/pytorch/pytorch/issues/13246#issuecomment... As you’ve pointed out fork() isn’t ideal for a number of reasons and in general it’s preferred to use torch tensors directly instead of numpy arrays so that you are not forced into using fork() There’s also this write up which I found to be quite useful for details: https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/ https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multip...

ahoka 3y ago

“I believe it's Copy-on-Write semantics, right? What happens when the parent process changes the array?” Yes and you just answered your question.

boppo1 3y ago
For a programming novice: Is CoW where a 'child' process operates on a copy of data stored elsewhere in memory, but you have some mechanism to always update the copy any time the parent process modifies/writes to the original data?
- porridgeandrice 3y ago
  No, if I have stuff, and I fork(), you, my child process is expected to have a copy of that stuff in it's current state. However, the kernel lies to you. It delays copying the stuff until I write to it i.e until I need to see a new state. Hence, "copy-on-write". In this case, I am the parent python process, you are the child processes started with the `multiprocessing` module, and the data is the numpy array. The implication is that the changes made by the parent process to the numpy array post fork() won't be visible to the child processes.