4 ms·

this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a hig

by slt2021 3y ago

this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws may argue this is egress for s3, just not for aws

yencabulator 2y ago
If they charged for the extents that are actually read from S3 (even if the network connection was just closed), that'd be fine. But charging for terabytes when the actual connection consumed a few kilobytes is still wrong; surely they didn't e.g. buffer those terabytes of data from S3 disk storage into memory. Charge for the buffer you fill: sure. Charge the theoretical maximum: silly.
vermilingua 3y ago
Then subsequent requests that hit the cache shouldn’t be charged by that logic.
- slt2021 3y ago
  s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet. if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch
  - jpambrun 2y ago
    But you are not charged if it's not Internet egress, which makes your argument about compute/ram cost moot.
jpambrun 2y ago
The consequence of your interpretation is that surprise billing is not even bounded by the physical limitations. This is absolutely not reasonable. I am the user with in the zip case. I knew this testing would incur a cost, but I could imagine it could be 20x my network speed and I have absolutely no control over what AWS claims to meter.

yencabulator 2y ago

If they charged for the extents that are actually read from S3 (even if the network connection was just closed), that'd be fine. But charging for terabytes when the actual connection consumed a few kilobytes is still wrong; surely they didn't e.g. buffer those terabytes of data from S3 disk storage into memory. Charge for the buffer you fill: sure. Charge the theoretical maximum: silly.

vermilingua 3y ago

Then subsequent requests that hit the cache shouldn’t be charged by that logic.

slt2021 3y ago
s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet. if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch
- jpambrun 2y ago
  But you are not charged if it's not Internet egress, which makes your argument about compute/ram cost moot.

jpambrun 2y ago

The consequence of your interpretation is that surprise billing is not even bounded by the physical limitations. This is absolutely not reasonable. I am the user with in the zip case. I knew this testing would incur a cost, but I could imagine it could be 20x my network speed and I have absolutely no control over what AWS claims to meter.