3 ms·
Possible. I had a fair amount of CPU going on, trying to keep 8 cores busy with SHA256(plaintext), encrypt, and SHA256(encrypted_blob). I was just trying to b
by sliken 4y ago
Possible. I had a fair amount of CPU going on, trying to keep 8 cores busy with SHA256(plaintext), encrypt, and SHA256(encrypted_blob). I was just trying to be straight forward, keep files queued, so that when a core went idle it could grab another.
The go was similarly very straight forward, walktree -> CheckIfNewOrchanged -> channel -> sha256/encrypt/sha256. Channels made it really easy/clear and performed quite well. I was getting near linear scaling, CPU time consumed was 8x the wall clock, and speed increased 7.9x or so. With python I was getting significantly less performance per core and worse scaling.
With 8 cores 10Gbit is 1.25GByte/8 = 160MB/sec which is ok, not great, depends on how much computation you are doing. My goal is keeping 100gbit saturated, but I am adding cores as well.
I do hope to compare to Go vs Rust as well.
- dekhn 4y agoChannels are just threadsafe queues with language support and an N:M threading model. Don't get me wrong; I agree it's easier to build performant applications in go, and to get the performance I want, I have to set my AWS boto3 S3 settings to have massive queues.