4 ms·
To preface this comment, I'm not super familiar with range GETs or the library you're using. Are you saying that the library was performing range GET requests
by tedmiston 2y ago
To preface this comment, I'm not super familiar with range GETs or the library you're using. Are you saying that the library was performing range GET requests but instead of sending a byte range, it was only sending a starting byte number?
> Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.
Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.
FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4 https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4
Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- jpambrun 2y agoYes. It made many request with Range star- (i.e. without end). 500$ is not making a huge dent in the monthly bill for this. I just got a few laughs at my expense from colleagues.. I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then. I could generate 10s of gbps of "apparent" billable traffic with a puny VPS.
- nicolas_17 2y agoYou could also generate 10s of gbps of actual billable traffic with a puny VPS by forging TCP ACKs and getting AWS to send you data faster than you can actually receive it, but that's closer to an actual DoS attack and more likely to get you in trouble...
- tedmiston 2y ago> I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then. I suppose that would be easy enough to test. I was curious how video streaming services handle the case where a stream is started or partially consumed. Mux meters it in terms of [seconds of] video segments delivered: > Cost is per minute of video delivered. To calculate video delivered, we measure the number of seconds of video delivered to a video player. Note that if a segment of video is delivered, it is charged, even if the viewer doesn’t actually watch the video. [1]: https://docs.mux.com/pricing/video#delivery https://docs.mux.com/pricing/video#delivery