6 ms·

I assume AWS will start using graviton for their S3, EBS, ELB etc services aggressively?

fh973 6y ago
The S3 protocol needs MD5 hashes, which these benchmarks don't cover. A Xeon core can do that with a couple of hundred MB/s. As it's a continuous hash of the whole object, this limits S3 performance. I guess AWS does not care that much about single stream S3 performance but will optimize for costs.
- justincormack 6y ago
  I don't think it uses md5 any more, the newer protocol versions support sha256 hashes and thats only if the user checks, may not be what is used internally.
  - chillaxtian 6y ago
    Link? The eTag of an object is still the MD5 AFAIK.
    - xxpor 6y ago
      The request signature (assuming version 4) is SHA256, the object eTag is MD5.
    - eadan 6y ago
      Yes, S3 uses the content-md5 header to verify data integrity for puts. https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjec...
    - rob-olmos 6y ago
      For the ETag, yes and no that it's an MD5 of the object depending on encryption and multipart upload: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRe... For the object upload, AWSv4 request signature uses SHA256 on the payload/object, but I don't know if S3 also computes & compares the digest or just uses the x-amz-content-sha256 header value.
      - rob-olmos 6y ago
        AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.
  - rob-olmos 6y ago
    Do you happen to know if AWSv4 with S3 also computes & compares the SHA256 of the payload/object or does it just use the specified x-amz-content-sha256 header's value? FYI S3 Object Lock requires Content-MD5 header.
    - rob-olmos 6y ago
      AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.
- cjhanks 6y ago
  Offloading a hash function to a custom chip seems appropriate. I suspect they are deploying an entire specialized box, not just a custom processor. Edit: Nevermind, I may have been thinking about the wrong end of the pipe.
- perrohunter 6y ago
  The MinIO code is highly optimized for the md5 hashes using AVX512 https://blog.min.io/accelerating-aggregate-md5-hashing-up-to-800-with-avx512-2/ https://blog.min.io/accelerating-aggregate-md5-hashing-up-to...
ec109685 6y ago
I believe they are already using them for ELB’s. It is mentioned in their announcement that they will, but I read somewhere else it was already happening: https://aws.amazon.com/blogs/aws/coming-soon-graviton2-powered-general-purpose-compute-optimized-memory-optimized-ec2-instances/ https://aws.amazon.com/blogs/aws/coming-soon-graviton2-power...
ksec 6y ago
They announced it is their intention to have all their services and SaaS running on top of Graviton instances. i.e They are eating their own dog food. They said this will be a long time, multi year method. ( And whenever Amazon is saying this that means they are doing as fast as possible to optimise for cost )

The S3 protocol needs MD5 hashes, which these benchmarks don't cover. A Xeon core can do that with a couple of hundred MB/s. As it's a continuous hash of the whole object, this limits S3 performance. I guess AWS does not care that much about single stream S3 performance but will optimize for costs.

justincormack 6y ago
I don't think it uses md5 any more, the newer protocol versions support sha256 hashes and thats only if the user checks, may not be what is used internally.
- chillaxtian 6y ago
  Link? The eTag of an object is still the MD5 AFAIK.
  - xxpor 6y ago
    The request signature (assuming version 4) is SHA256, the object eTag is MD5.
  - eadan 6y ago
    Yes, S3 uses the content-md5 header to verify data integrity for puts. https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjec...
  - rob-olmos 6y ago
    For the ETag, yes and no that it's an MD5 of the object depending on encryption and multipart upload: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRe... For the object upload, AWSv4 request signature uses SHA256 on the payload/object, but I don't know if S3 also computes & compares the digest or just uses the x-amz-content-sha256 header value.
    - rob-olmos 6y ago
      AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.
- rob-olmos 6y ago
  Do you happen to know if AWSv4 with S3 also computes & compares the SHA256 of the payload/object or does it just use the specified x-amz-content-sha256 header's value? FYI S3 Object Lock requires Content-MD5 header.
  - rob-olmos 6y ago
    AWS support via Twitter said that the SHA256 digest of the payload is calculated and compared for S3 when using AWSv4 signature requests.
cjhanks 6y ago
Offloading a hash function to a custom chip seems appropriate. I suspect they are deploying an entire specialized box, not just a custom processor. Edit: Nevermind, I may have been thinking about the wrong end of the pipe.
perrohunter 6y ago
The MinIO code is highly optimized for the md5 hashes using AVX512 https://blog.min.io/accelerating-aggregate-md5-hashing-up-to-800-with-avx512-2/ https://blog.min.io/accelerating-aggregate-md5-hashing-up-to...

ec109685 6y ago

I believe they are already using them for ELB’s. It is mentioned in their announcement that they will, but I read somewhere else it was already happening: https://aws.amazon.com/blogs/aws/coming-soon-graviton2-powered-general-purpose-compute-optimized-memory-optimized-ec2-instances/ https://aws.amazon.com/blogs/aws/coming-soon-graviton2-power...

ksec 6y ago

They announced it is their intention to have all their services and SaaS running on top of Graviton instances. i.e They are eating their own dog food. They said this will be a long time, multi year method. ( And whenever Amazon is saying this that means they are doing as fast as possible to optimise for cost )