3 ms·

Their CLI manages to completely lock up my up to date MacBook Pro when simply downloading files and has very strange and conflicting choices of parameters. It i

Their CLI manages to completely lock up my up to date MacBook Pro when simply downloading files and has very strange and conflicting choices of parameters. It is comprehensive, but I wouldn’t call it good

acdha 7y ago
I won’t defend their CLI arguments other than noting that they usually closely follow the also poorly-named APIs[1], but locking up is highly unusual — do you have something like AV software or other quasi-malware which might be interfering with normal I/O? We’ve used it for many millions of files over the years and that’s never been an issue on macOS or Linux. 1. AWS needs a Chief Consistency Officer who can block shipping until you cleanup the prototype slop
- llimllib 7y ago
  Doing an ‘aws s3 sync ...’ on a directory with large files causes 100% CPU usage
  - acdha 7y ago
    How would you compare hashes without calculating them? Any operating system more advanced than Windows 95 shouldn’t “lock up” with a CPU-bound task.
    - llimllib 7y ago
      An extremely naive program can sha1 hash 1 million 100 byte strings on my computer in less than half a second: https://gist.github.com/llimllib/72f60aa33b32e422962d876ddf0a4660 https://gist.github.com/llimllib/72f60aa33b32e422962d876ddf0... This is literally the first program I came up with, no attempt to optimize it at all. There is zero chance that the AWS sync command is filling my CPU just by hashing bytes edit: I'm going to try not to let you nerd snipe me into doing the profiling the AWS CLI needs to be doing, for them. Because that's now what I desire to do.
      - ptomato 7y ago
        so 200 megabytes/second? I'm not sure what your definition of large files is, but hashing anything sizable with SHA1 is trivially CPU-bound with any modern SSD, in the absence of a processor with the sha asm extensions. that being said, quick glance at the source suggests that awscli's s3 sync only compares files by size & timestamp, not etag, so it's not hashing anything client-side.
nickjj 7y ago
I can't say I've had the same experience here on Windows using WSL. It runs like a champ. I've also used it within Debian based Docker images on many occasions.
fortran77 7y ago
If your MacBook is truly "locking up" and needs a hard reboot, I doubt the problem is the CLI.
- llimllib 7y ago
  It pegs the processor, blocking UI updates. A reboot will not help that

acdha 7y ago

I won’t defend their CLI arguments other than noting that they usually closely follow the also poorly-named APIs[1], but locking up is highly unusual — do you have something like AV software or other quasi-malware which might be interfering with normal I/O? We’ve used it for many millions of files over the years and that’s never been an issue on macOS or Linux. 1. AWS needs a Chief Consistency Officer who can block shipping until you cleanup the prototype slop

llimllib 7y ago
Doing an ‘aws s3 sync ...’ on a directory with large files causes 100% CPU usage
- acdha 7y ago
  How would you compare hashes without calculating them? Any operating system more advanced than Windows 95 shouldn’t “lock up” with a CPU-bound task.
  - llimllib 7y ago
    An extremely naive program can sha1 hash 1 million 100 byte strings on my computer in less than half a second: https://gist.github.com/llimllib/72f60aa33b32e422962d876ddf0a4660 https://gist.github.com/llimllib/72f60aa33b32e422962d876ddf0... This is literally the first program I came up with, no attempt to optimize it at all. There is zero chance that the AWS sync command is filling my CPU just by hashing bytes edit: I'm going to try not to let you nerd snipe me into doing the profiling the AWS CLI needs to be doing, for them. Because that's now what I desire to do.
    - ptomato 7y ago
      so 200 megabytes/second? I'm not sure what your definition of large files is, but hashing anything sizable with SHA1 is trivially CPU-bound with any modern SSD, in the absence of a processor with the sha asm extensions. that being said, quick glance at the source suggests that awscli's s3 sync only compares files by size & timestamp, not etag, so it's not hashing anything client-side.

nickjj 7y ago

I can't say I've had the same experience here on Windows using WSL. It runs like a champ. I've also used it within Debian based Docker images on many occasions.

fortran77 7y ago

If your MacBook is truly "locking up" and needs a hard reboot, I doubt the problem is the CLI.

llimllib 7y ago
It pegs the processor, blocking UI updates. A reboot will not help that