5 ms·

So I guess the attack of the OP is a case AWS calculate price based on the unbounded request header and not on the actual egress

scottlamb 2y ago
More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article goes on to say: > Okay, this is half the explanation. AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. In other words, how much they bill really isn't bounded by how much is sent at all. This is unacceptable. [1] https://en.wikipedia.org/wiki/TCP_window_scale_option https://en.wikipedia.org/wiki/TCP_window_scale_option
- easton 2y ago
  > this completely fails to explain the article's observed 3 TB sent vs 130 TB billed I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.
  - scottlamb 2y ago
    Even so, TCP windows generally don't start at the maximum size, so I'm really not buying that ratio of bytes ACKed to sent. One could disprove this explanation with a packet capture; just add the remaining window.
- slt2021 2y ago
  this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws may argue this is egress for s3, just not for aws
  - yencabulator 2y ago
    If they charged for the extents that are actually read from S3 (even if the network connection was just closed), that'd be fine. But charging for terabytes when the actual connection consumed a few kilobytes is still wrong; surely they didn't e.g. buffer those terabytes of data from S3 disk storage into memory. Charge for the buffer you fill: sure. Charge the theoretical maximum: silly.
  - vermilingua 2y ago
    Then subsequent requests that hit the cache shouldn’t be charged by that logic.
    - slt2021 2y ago
      s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet. if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch
      - jpambrun 2y ago
        But you are not charged if it's not Internet egress, which makes your argument about compute/ram cost moot.
  - jpambrun 2y ago
    The consequence of your interpretation is that surprise billing is not even bounded by the physical limitations. This is absolutely not reasonable. I am the user with in the zip case. I knew this testing would incur a cost, but I could imagine it could be 20x my network speed and I have absolutely no control over what AWS claims to meter.
- paulddraper 2y ago
  > AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. But egress fees only apply to S3 transfers outside AWS? So which is it? Data transferred to the Internet? Or data processed internally?
  - scottlamb 2y ago
    Obviously they break the problem down into questions like "what's the maximum that could be transferred between any two components? what's the most expensive components this transits?" and pick the most expensive answers without any attempt at coherence. Right now some class action lawyer is drooling.

scottlamb 2y ago

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article goes on to say: > Okay, this is half the explanation. AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. In other words, how much they bill really isn't bounded by how much is sent at all. This is unacceptable. [1] https://en.wikipedia.org/wiki/TCP_window_scale_option https://en.wikipedia.org/wiki/TCP_window_scale_option

easton 2y ago
> this completely fails to explain the article's observed 3 TB sent vs 130 TB billed I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.
- scottlamb 2y ago
  Even so, TCP windows generally don't start at the maximum size, so I'm really not buying that ratio of bytes ACKed to sent. One could disprove this explanation with a packet capture; just add the remaining window.
slt2021 2y ago
this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws may argue this is egress for s3, just not for aws
- yencabulator 2y ago
  If they charged for the extents that are actually read from S3 (even if the network connection was just closed), that'd be fine. But charging for terabytes when the actual connection consumed a few kilobytes is still wrong; surely they didn't e.g. buffer those terabytes of data from S3 disk storage into memory. Charge for the buffer you fill: sure. Charge the theoretical maximum: silly.
- vermilingua 2y ago
  Then subsequent requests that hit the cache shouldn’t be charged by that logic.
  - slt2021 2y ago
    s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet. if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch
    - jpambrun 2y ago
      But you are not charged if it's not Internet egress, which makes your argument about compute/ram cost moot.
- jpambrun 2y ago
  The consequence of your interpretation is that surprise billing is not even bounded by the physical limitations. This is absolutely not reasonable. I am the user with in the zip case. I knew this testing would incur a cost, but I could imagine it could be 20x my network speed and I have absolutely no control over what AWS claims to meter.
paulddraper 2y ago
> AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. But egress fees only apply to S3 transfers outside AWS? So which is it? Data transferred to the Internet? Or data processed internally?
- scottlamb 2y ago
  Obviously they break the problem down into questions like "what's the maximum that could be transferred between any two components? what's the most expensive components this transits?" and pick the most expensive answers without any attempt at coherence. Right now some class action lawyer is drooling.