5 ms·

>Try process level I/O, such pipes, sockets, and the like. This. > Have Linux deal with the concurrency problem, not you. Not just Linux. We did this with o

>Try process level I/O, such pipes, sockets, and the like. This. > Have Linux deal with the concurrency problem, not you. Not just Linux. We did this with our Windows app rewrite. IPC with pipes is fast as hell, just works, and it greatly simplified parallelism for us.

spacechild1 4y ago
But why do you need a seperate process? You can do the same with threads and queues. The only advantage I can think of is sandboxing, i.e. preventing a misbehaving task from taking down your whole app.
- JustSomeNobody 4y ago
  Ease of development. Ease of maintenance. Separation of concerns. Just to name a few. For example, one service is for hardware communication. It's job is to communicate with all the various devices, and forward those messages out. >You can do the same with threads and queues. Yes, but as OP up the chain mentions, you can simplify by letting the OS handle some of that.
  - spacechild1 4y ago
    > Ease of development. Ease of maintenance. Seperation of concerns. I'm not convinced. Managing processes is much more complex than managing threads, particularly if the code should be cross-platform. You need a very good library to hide away all the nasty OS specific details. Same goes for pipes or sockets. Then there is the whole issue of message (de)serialization. I don't see how this can possibly be easier than starting a thread and communicating with concurrent queue. I mean, subprocesses certainly have their use cases, but I would never see them as a drop-in replacement for threads. > you can simplify by letting the OS handle some of that. Processes and threads are both OS resources. For the OS scheduler they are practically equivalent.
    - dragontamer 4y ago
      Consider: what is the easiest way to parallelize GCC to a 32 core system? Answer: make -j128. Way easier than trying to make every data structure inside of the compiler into a parallel programming model. -------- Process level parallelism is a higher level of thinking. As long as you have spare RAM (and let's be frank, we all have 32GB+ sitting around these days), you can add more and more processes to solve your problem. If you are talking data structures and mutexes, you are working at a far more complex layer than what I'm saying for #1. ---- Even completely single threaded code can be run in parallel in many cases in practice, because we have multiple files or other divisions of labor available at the process / user level.
      - spacechild1 4y ago
        > Consider: what is the easiest way to parallelize GCC to a 32 core system? > Answer: make -j128. > Way easier than trying to make every data structure inside of the compiler into a parallel programming model. That's a false dichotomy. If the compiler program was implemented as a library, I would rather create 128 threads that each call compileSourceFile() than spawn 128 subprocesses that do the same thing. The question you should be asking is: do I need my task to execute in a seperate address space? If yes, spawn a subprocess, otherwise use a thread.
        dragontamer 4y ago
        > I would rather create 128 threads that each call compileSourceFile() than spawn 128 subprocesses that do the same thing. Would you rather write "compileSourceFile()" in a reentrant way (ie: no global variables, no static variables, guaranteed reentrancy, and other such requirements to work in a typical pthread manner)... or would you rather have processes where all those things are fine and not bugs? The minute you start up threads with implicitly shared memory spaces... the minute "singleton pattern" suddenly grows complex. > The question you should be asking is: do I need my task to execute in a seperate address space? If yes, spawn a subprocess, otherwise use a thread. On the contrary. Separate address spaces by default is far easier. Thread#45 going crazy due to buffer-overflows will demolish thread#25. But process#45 with a buffer-overflow will not affect process#25. I/O is also grossly simplified inside the process model. Closing out a process closes() all sockets, pipes, and file I/O automatically, no matter how the process dies. (Ex: Segfaults, kill -9, etc. etc. are all handled gracefully). If one thread dies, for whatever reason, your program is extremely hosed. Its very difficult to reason where the legitimate state of your multithreaded data-structures is in. Threads are far more efficient, yes. So if you need efficiency, use them. But most people in my experience are Python or PHP programmers (or other such high level language), where it is clear that performance isn't an issue.

spacechild1 4y ago

But why do you need a seperate process? You can do the same with threads and queues. The only advantage I can think of is sandboxing, i.e. preventing a misbehaving task from taking down your whole app.

JustSomeNobody 4y ago
Ease of development. Ease of maintenance. Separation of concerns. Just to name a few. For example, one service is for hardware communication. It's job is to communicate with all the various devices, and forward those messages out. >You can do the same with threads and queues. Yes, but as OP up the chain mentions, you can simplify by letting the OS handle some of that.
- spacechild1 4y ago
  > Ease of development. Ease of maintenance. Seperation of concerns. I'm not convinced. Managing processes is much more complex than managing threads, particularly if the code should be cross-platform. You need a very good library to hide away all the nasty OS specific details. Same goes for pipes or sockets. Then there is the whole issue of message (de)serialization. I don't see how this can possibly be easier than starting a thread and communicating with concurrent queue. I mean, subprocesses certainly have their use cases, but I would never see them as a drop-in replacement for threads. > you can simplify by letting the OS handle some of that. Processes and threads are both OS resources. For the OS scheduler they are practically equivalent.
  - dragontamer 4y ago
    Consider: what is the easiest way to parallelize GCC to a 32 core system? Answer: make -j128. Way easier than trying to make every data structure inside of the compiler into a parallel programming model. -------- Process level parallelism is a higher level of thinking. As long as you have spare RAM (and let's be frank, we all have 32GB+ sitting around these days), you can add more and more processes to solve your problem. If you are talking data structures and mutexes, you are working at a far more complex layer than what I'm saying for #1. ---- Even completely single threaded code can be run in parallel in many cases in practice, because we have multiple files or other divisions of labor available at the process / user level.
    - spacechild1 4y ago
      > Consider: what is the easiest way to parallelize GCC to a 32 core system? > Answer: make -j128. > Way easier than trying to make every data structure inside of the compiler into a parallel programming model. That's a false dichotomy. If the compiler program was implemented as a library, I would rather create 128 threads that each call compileSourceFile() than spawn 128 subprocesses that do the same thing. The question you should be asking is: do I need my task to execute in a seperate address space? If yes, spawn a subprocess, otherwise use a thread.
      - dragontamer 4y ago
        > I would rather create 128 threads that each call compileSourceFile() than spawn 128 subprocesses that do the same thing. Would you rather write "compileSourceFile()" in a reentrant way (ie: no global variables, no static variables, guaranteed reentrancy, and other such requirements to work in a typical pthread manner)... or would you rather have processes where all those things are fine and not bugs? The minute you start up threads with implicitly shared memory spaces... the minute "singleton pattern" suddenly grows complex. > The question you should be asking is: do I need my task to execute in a seperate address space? If yes, spawn a subprocess, otherwise use a thread. On the contrary. Separate address spaces by default is far easier. Thread#45 going crazy due to buffer-overflows will demolish thread#25. But process#45 with a buffer-overflow will not affect process#25. I/O is also grossly simplified inside the process model. Closing out a process closes() all sockets, pipes, and file I/O automatically, no matter how the process dies. (Ex: Segfaults, kill -9, etc. etc. are all handled gracefully). If one thread dies, for whatever reason, your program is extremely hosed. Its very difficult to reason where the legitimate state of your multithreaded data-structures is in. Threads are far more efficient, yes. So if you need efficiency, use them. But most people in my experience are Python or PHP programmers (or other such high level language), where it is clear that performance isn't an issue.