3 ms·

I did the benchmark and multiprocessing achieved higher bandwidth than multithreading. 8 times higher. This was just reading from disk, so no hashing step at a

I did the benchmark and multiprocessing achieved higher bandwidth than multithreading. 8 times higher. This was just reading from disk, so no hashing step at all. Didn't even write to disk. I think you might not be so knowledgable about concurrency.

Geminidog 6y ago
Did you read my full post? I don't think you did. Let me quote it: >To make it a fair test and to see the benefits of python threads over processes you need to get rid of the hash operation. Maybe make your python program write a copy the file to another file. Then try to make everything concurrent. By everything I mean for every single file, fire off a new thread and do the same for processes. >Though the other bottleneck will be your HD in this case. When your HDs see 200000 IO calls the HD's themselves will start serializing the requests. Also with such short message flight time in the wire is super short. Likely your IO is spending a good chunk of time writing messages to program memory as well. >The most fair test is 200000 IO calls to multiple external services that can handle such volume. Don't touch your HD. Just focus tests on services that will not block or serialize IO. I'm very very knowledgeable about concurrency. The problem is you. You failed to read my post and the caveats of all the details I mentioned. You failed to adjust the test fully to make it fair. You just overall failed and you don't know much. Let me explain why you failed. Like I quoted above. HD IO has very very short time of flight meaning that the actual operation of compute (writing the data to program memory) takes much more time. Your threads aren't parallelizing that. Like I said before THE ONLY DIFFERENCE in processes and threads is that python threads DO NOT parallelize compute and processes are MORE EXPENSIVE to spawn in terms of memory space. SO if you logically parallelize off of a fix number of threads/processes what do you think will occur? Of course processes will be faster. I told you to adjust the test to spawn a new thread/process for every IO call to see the benefits. Because threads are cheaper in memory to spawn it will be able to spawn MANY more threads than it can processes. So if your operation is IO bound what you typically did with like 10 processes can likely be done with 1000 threads. <--- That is the difference. You said you have 2000000 files? That means 2000000 threads/processes must be spawned for a fair test. And if you can, try to hit some server on the internet that can handle that many requests to fully see parallelization of IO because time of flight is longer over the internet. Additionally your HDD's themselves can't parallelize 2000000 requests so in the end all your threads/processes will be bottlenecked by the HD itself NOT by IO. That's why I told you to hit something over the internet that can handle 2000000 IO calls. I mentioned All of this in my original response and you failed to understand. Now do you Get it? I actually am trying to teach you something and you have the balls to tell me that I don't know about concurrency. It's quite obvious you have no idea what you're talking about.
- jacobwilliamroy 6y ago
  If I was doing network I/O then there's a chance that threads would be faster... but you do realize that would also be multiprocessing on separate physical CPUs right? A separate computer, running a separate process, is processing your request on the network at the same time. That's how async works. It's really just an illusion of single-thread paralellism. Anyways, the original scenario didn't require any network services and hashing each data chunk and writing to ramdisk really was instant. My script originally was supposed to enable integrity checks and deduplication of files on separate hard drives. There's no point turning it into a webapp just to shoehorn async in there. Also this is a dumb conversation. I didn't actually run the benchmark. I just lied to you because I wanted to see you waste hours of your life typing out a wall of text for someone who doesn't care what you think. Also I still think you have no idea what you're talking about.
  - Geminidog 6y ago
    >Also this is a dumb conversation. I didn't actually run the benchmark. I just lied to you because I wanted to see you waste hours of your life typing out a wall of text for someone who doesn't care what you think. Also I still think you have no idea what you're talking about. I was trying out of my heart to help out someone who didn't know anything. Looks like you took my altruism and threw it on the ground and stepped on it. >If I was doing network I/O then there's a chance that threads would be faster... but you do realize that would also be multiprocessing on separate physical CPUs right? Your HDD has it's own processor inside of it that handles reads and writes. Your CPU only sends it commands. So it's the same thing either way. It's just servers are designed to be hit by 200000 simultaneous requests, HDDs are not. >That's how async works. It's really just an illusion of single-thread paralellism. Except IO messages travel down the wire in parallel. This is what you need to get through your head. For compute, time is divided among threads in a single core but not for IO. >Anyways, the original scenario didn't require any network services and hashing each data chunk and writing to ramdisk really was instant. My script originally was supposed to enable integrity checks and deduplication of files on separate hard drives. There's no point turning it into a webapp just to shoehorn async in there. I didn't tell you to turn it into a webapp. I told you that your test didn't make sense and I told you to run a completely different test. I didn't tell you to shoehorn your app. If you're using sockets the API for interfacing with IO is exactly the same whether it's web OR an HDD. No shoehorning. HDD still can benefit from async.

Geminidog 6y ago

Did you read my full post? I don't think you did. Let me quote it: >To make it a fair test and to see the benefits of python threads over processes you need to get rid of the hash operation. Maybe make your python program write a copy the file to another file. Then try to make everything concurrent. By everything I mean for every single file, fire off a new thread and do the same for processes. >Though the other bottleneck will be your HD in this case. When your HDs see 200000 IO calls the HD's themselves will start serializing the requests. Also with such short message flight time in the wire is super short. Likely your IO is spending a good chunk of time writing messages to program memory as well. >The most fair test is 200000 IO calls to multiple external services that can handle such volume. Don't touch your HD. Just focus tests on services that will not block or serialize IO. I'm very very knowledgeable about concurrency. The problem is you. You failed to read my post and the caveats of all the details I mentioned. You failed to adjust the test fully to make it fair. You just overall failed and you don't know much. Let me explain why you failed. Like I quoted above. HD IO has very very short time of flight meaning that the actual operation of compute (writing the data to program memory) takes much more time. Your threads aren't parallelizing that. Like I said before THE ONLY DIFFERENCE in processes and threads is that python threads DO NOT parallelize compute and processes are MORE EXPENSIVE to spawn in terms of memory space. SO if you logically parallelize off of a fix number of threads/processes what do you think will occur? Of course processes will be faster. I told you to adjust the test to spawn a new thread/process for every IO call to see the benefits. Because threads are cheaper in memory to spawn it will be able to spawn MANY more threads than it can processes. So if your operation is IO bound what you typically did with like 10 processes can likely be done with 1000 threads. <--- That is the difference. You said you have 2000000 files? That means 2000000 threads/processes must be spawned for a fair test. And if you can, try to hit some server on the internet that can handle that many requests to fully see parallelization of IO because time of flight is longer over the internet. Additionally your HDD's themselves can't parallelize 2000000 requests so in the end all your threads/processes will be bottlenecked by the HD itself NOT by IO. That's why I told you to hit something over the internet that can handle 2000000 IO calls. I mentioned All of this in my original response and you failed to understand. Now do you Get it? I actually am trying to teach you something and you have the balls to tell me that I don't know about concurrency. It's quite obvious you have no idea what you're talking about.

jacobwilliamroy 6y ago
If I was doing network I/O then there's a chance that threads would be faster... but you do realize that would also be multiprocessing on separate physical CPUs right? A separate computer, running a separate process, is processing your request on the network at the same time. That's how async works. It's really just an illusion of single-thread paralellism. Anyways, the original scenario didn't require any network services and hashing each data chunk and writing to ramdisk really was instant. My script originally was supposed to enable integrity checks and deduplication of files on separate hard drives. There's no point turning it into a webapp just to shoehorn async in there. Also this is a dumb conversation. I didn't actually run the benchmark. I just lied to you because I wanted to see you waste hours of your life typing out a wall of text for someone who doesn't care what you think. Also I still think you have no idea what you're talking about.
- Geminidog 6y ago
  >Also this is a dumb conversation. I didn't actually run the benchmark. I just lied to you because I wanted to see you waste hours of your life typing out a wall of text for someone who doesn't care what you think. Also I still think you have no idea what you're talking about. I was trying out of my heart to help out someone who didn't know anything. Looks like you took my altruism and threw it on the ground and stepped on it. >If I was doing network I/O then there's a chance that threads would be faster... but you do realize that would also be multiprocessing on separate physical CPUs right? Your HDD has it's own processor inside of it that handles reads and writes. Your CPU only sends it commands. So it's the same thing either way. It's just servers are designed to be hit by 200000 simultaneous requests, HDDs are not. >That's how async works. It's really just an illusion of single-thread paralellism. Except IO messages travel down the wire in parallel. This is what you need to get through your head. For compute, time is divided among threads in a single core but not for IO. >Anyways, the original scenario didn't require any network services and hashing each data chunk and writing to ramdisk really was instant. My script originally was supposed to enable integrity checks and deduplication of files on separate hard drives. There's no point turning it into a webapp just to shoehorn async in there. I didn't tell you to turn it into a webapp. I told you that your test didn't make sense and I told you to run a completely different test. I didn't tell you to shoehorn your app. If you're using sockets the API for interfacing with IO is exactly the same whether it's web OR an HDD. No shoehorning. HDD still can benefit from async.