5 ms·

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer

by wmf 3y ago

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer of the entire file is billed." WTF if true.

fabian2k 3y ago
That sounds egregious enough that I have trouble believing this can be correct. My understanding is that AWS bills for egress for every service, parts of the file that aren't transferred are not part of this so can't be billed. There could certainly be S3-specific charges that affect cases like this, no idea. But if AWS bills the full egress traffic costs for a range request I'd consider that essentially fraud.
- belter 3y ago
  https://github.com/ZJONSSON/node-unzipper/issues/308 https://github.com/ZJONSSON/node-unzipper/issues/308
  - paulddraper 3y ago
    tl;dr AWS user believes that testing on a 1Gbps connection for 45 min can't be more than $10 of egress. Gets a $500 bill instead. Note: This user specified a lower range but not an upper range on the request (and closed the connection prematurely). Essentially read() with an offset, for a ZIP tool. See also: https://news.ycombinator.com/item?id=40205213 https://news.ycombinator.com/item?id=40205213
    - yonixwm 3y ago
      So I guess the attack of the OP is a case AWS calculate price based on the unbounded request header and not on the actual egress
      - scottlamb 3y ago
        More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article goes on to say: > Okay, this is half the explanation. AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. In other words, how much they bill really isn't bounded by how much is sent at all. This is unacceptable. [1] https://en.wikipedia.org/wiki/TCP_window_scale_option https://en.wikipedia.org/wiki/TCP_window_scale_option
        easton 3y ago
        > this completely fails to explain the article's observed 3 TB sent vs 130 TB billed I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.
        scottlamb 3y ago
        Even so, TCP windows generally don't start at the maximum size, so I'm really not buying that ratio of bytes ACKed to sent. One could disprove this explanation with a packet capture; just add the remaining window.
        slt2021 3y ago
        this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws may argue this is egress for s3, just not for aws
    - fabian2k 3y ago
      There is probably a small area where it's difficult to measure, so I would not expect billing to be exact to the byte here. But billing for the requested range if not the entire range was actually transferred is just not correct and not acceptable.
      - paulddraper 3y ago
        Certainly if you are charging for internet egress. Like, charging for requests or internal data processing, sure okay. But this is a charge specifically for the data transferred from AWS to the internet. So if you're not transferring data to the internet....
        fabian2k 3y ago
        The part where I think there is some flexibility is about the difference between "bytes attempted to transfer" and "bytes actually transferred". I think it is pretty fair to bill for the former, as long as you abort requests in a reasonable way. So I don't expect it to be billed exactly by the transferred byte, but I do expect it to not go above that higher than whatever the chunk size for transferring is.
        paulddraper 3y ago
        Sure. In this case specifically AWS is attempting to transfer 70Gbps through a 1Gbps pipe. Not a rounding error.
        deleted 3y ago
        [deleted]
        klabb3 3y ago
        That’s an orthogonal issue. There’s no interpretation of “egress” that means “stuff we do internally before leaving aws data centers”. If the tcp conn is reset only a few MB would leave aws frontend servers. Instead, it appears they’ve been basing the number off the range in the request and/or whatever internal caching/loading they’re doing within S3, which again has nothing to do with egress. I mean, we already know egress is short for egregious. It’s an incredibly bad look to be overestimating the “fuck you” part of the bill.
- __roland__ 3y ago
  Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we received for this is that it's due to some internal buffering S3 is doing, and they do count this as egress). In any case, if you send and cancel such requests quickly (which is easy enough, this was not even an adversarial situation, just a bug in some client API code) the egress cost is many times higher than your theoretical bandwidth (and about 80x higher than in the AWS documentation, hence the blogpost).
  - nijave 3y ago
    This is a problem with lots of services. Blocking large quantities of legitimate looking requests is a hard problem. Request cancellation is also tricky and not supported well in a lot of frameworks/programming languages.
    - thayne 3y ago
      But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.
      - martin-lim 3y ago
        Sadly this is no bug. Customers are NOT billed for actual data transferred. Customers are billed for "some kind of" data requested. If you interrupt the data transfer before downloading all requested data then it's "your fault". AWS documents this detail somewhere hidden on the S3 pricing page (https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/). Search for "Data Transfer Out may be different from the data received" in the "Data transfer" tab.
nicklecompte 3y ago
This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem compared to EC2. It also seemed like the root cause was an interrupted range request (although I wasn't fully clear on that). Even so that seems like a recent regression. It took me ages to get that stupid app working, I interrupted a lot of range requests :)
- __roland__ 3y ago
  You are right, this is about canceling range requests and still getting billed, not about requesting ranges and getting billed for the complete file egress. Sorry; we'll make the post clearer.
  - nextaccountic 3y ago
    This seems like something that customers can potentially sue for. AWS is overcharging them and not correcting the bug once they become aware of it.
- nielsole 3y ago
  S3 egress costs are free if the traffic stays within AWS. Sounds like your clients were EC2 instances so this wouldn't apply to you, would it?
  - mikepurvis 3y ago
    If it was a web application as stated in the GP, then it would indeed be egress as the request would be coming from a browser.
    - nicklecompte 3y ago
      Yes, it was client-side JavaScript making the range requests, asking for a string of genomic data to render in the browser. It was only to give the scientists a pretty picture :) The EC2 costs were largely ElasticSearch for a different function, which never looked at the data in S3.

by wmf 3y ago

fabian2k 3y ago

That sounds egregious enough that I have trouble believing this can be correct. My understanding is that AWS bills for egress for every service, parts of the file that aren't transferred are not part of this so can't be billed. There could certainly be S3-specific charges that affect cases like this, no idea. But if AWS bills the full egress traffic costs for a range request I'd consider that essentially fraud.

belter 3y ago
https://github.com/ZJONSSON/node-unzipper/issues/308 https://github.com/ZJONSSON/node-unzipper/issues/308
- paulddraper 3y ago
  tl;dr AWS user believes that testing on a 1Gbps connection for 45 min can't be more than $10 of egress. Gets a $500 bill instead. Note: This user specified a lower range but not an upper range on the request (and closed the connection prematurely). Essentially read() with an offset, for a ZIP tool. See also: https://news.ycombinator.com/item?id=40205213 https://news.ycombinator.com/item?id=40205213
  - yonixwm 3y ago
    So I guess the attack of the OP is a case AWS calculate price based on the unbounded request header and not on the actual egress
    - scottlamb 3y ago
      More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article goes on to say: > Okay, this is half the explanation. AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. In other words, how much they bill really isn't bounded by how much is sent at all. This is unacceptable. [1] https://en.wikipedia.org/wiki/TCP_window_scale_option https://en.wikipedia.org/wiki/TCP_window_scale_option
      - easton 3y ago
        > this completely fails to explain the article's observed 3 TB sent vs 130 TB billed I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.
        scottlamb 3y ago
        Even so, TCP windows generally don't start at the maximum size, so I'm really not buying that ratio of bytes ACKed to sent. One could disprove this explanation with a packet capture; just add the remaining window.
      - slt2021 3y ago
        this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws may argue this is egress for s3, just not for aws
  - fabian2k 3y ago
    There is probably a small area where it's difficult to measure, so I would not expect billing to be exact to the byte here. But billing for the requested range if not the entire range was actually transferred is just not correct and not acceptable.
    - paulddraper 3y ago
      Certainly if you are charging for internet egress. Like, charging for requests or internal data processing, sure okay. But this is a charge specifically for the data transferred from AWS to the internet. So if you're not transferring data to the internet....
      - fabian2k 3y ago
        The part where I think there is some flexibility is about the difference between "bytes attempted to transfer" and "bytes actually transferred". I think it is pretty fair to bill for the former, as long as you abort requests in a reasonable way. So I don't expect it to be billed exactly by the transferred byte, but I do expect it to not go above that higher than whatever the chunk size for transferring is.
        paulddraper 3y ago
        Sure. In this case specifically AWS is attempting to transfer 70Gbps through a 1Gbps pipe. Not a rounding error.
        deleted 3y ago
        [deleted]
        klabb3 3y ago
        That’s an orthogonal issue. There’s no interpretation of “egress” that means “stuff we do internally before leaving aws data centers”. If the tcp conn is reset only a few MB would leave aws frontend servers. Instead, it appears they’ve been basing the number off the range in the request and/or whatever internal caching/loading they’re doing within S3, which again has nothing to do with egress. I mean, we already know egress is short for egregious. It’s an incredibly bad look to be overestimating the “fuck you” part of the bill.
__roland__ 3y ago
Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we received for this is that it's due to some internal buffering S3 is doing, and they do count this as egress). In any case, if you send and cancel such requests quickly (which is easy enough, this was not even an adversarial situation, just a bug in some client API code) the egress cost is many times higher than your theoretical bandwidth (and about 80x higher than in the AWS documentation, hence the blogpost).
- nijave 3y ago
  This is a problem with lots of services. Blocking large quantities of legitimate looking requests is a hard problem. Request cancellation is also tricky and not supported well in a lot of frameworks/programming languages.
  - thayne 3y ago
    But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.
    - martin-lim 3y ago
      Sadly this is no bug. Customers are NOT billed for actual data transferred. Customers are billed for "some kind of" data requested. If you interrupt the data transfer before downloading all requested data then it's "your fault". AWS documents this detail somewhere hidden on the S3 pricing page (https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/). Search for "Data Transfer Out may be different from the data received" in the "Data transfer" tab.

nicklecompte 3y ago

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem compared to EC2. It also seemed like the root cause was an interrupted range request (although I wasn't fully clear on that). Even so that seems like a recent regression. It took me ages to get that stupid app working, I interrupted a lot of range requests :)

__roland__ 3y ago
You are right, this is about canceling range requests and still getting billed, not about requesting ranges and getting billed for the complete file egress. Sorry; we'll make the post clearer.
- nextaccountic 3y ago
  This seems like something that customers can potentially sue for. AWS is overcharging them and not correcting the bug once they become aware of it.
nielsole 3y ago
S3 egress costs are free if the traffic stays within AWS. Sounds like your clients were EC2 instances so this wouldn't apply to you, would it?
- mikepurvis 3y ago
  If it was a web application as stated in the GP, then it would indeed be egress as the request would be coming from a browser.
  - nicklecompte 3y ago
    Yes, it was client-side JavaScript making the range requests, asking for a string of genomic data to render in the browser. It was only to give the scientists a pretty picture :) The EC2 costs were largely ElasticSearch for a different function, which never looked at the data in S3.