4 ms·
No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than
by gymbeaux 2y ago
No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than what it was.
If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size.
It’s discussed here, although they’re talking about it being “CLI vs console” which may be true?
https://stackoverflow.com/questions/57201659/s3-bucket-size-differs-while-using-console-and-cli https://stackoverflow.com/questions/57201659/s3-bucket-size-...
- abadpoli 2y agoThis almost certainly has nothing to do with “saving CPU cycles” and is most likely just that whoever created the Cloudwatch console used the same rounding that is used for all other metrics in CW, rather than the proper calculation for disk size, and it was a small enough issue that it was never caught until it was too late to change it because changing it would disrupt the customers that have gotten used to it.
- presentation 2y agoIf anything a bit shift could do the 1024 division faster
- DougBTX 2y agoS3 doesn't format the size for human display, the object's Size is returned in bytes: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...
- gymbeaux 2y agoWhat I meant to say was S3 uses 2^20 to convert from bytes to kilobytes rather than 10^6 (e.g. what Windows uses). https://repost.aws/knowledge-center/s3-console-metric-discrepancy https://repost.aws/knowledge-center/s3-console-metric-discre... > 3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000)
- Self-Perfection 2y agoBut KB is indeed 1000 bytes, and MB is indeed 1000 KB. In case of 2^10 units the correct names are kibibyte (KiB) and mebibyte (MiB). Check https://en.wikipedia.org/wiki/Mebibyte#Multiple-byte_units https://en.wikipedia.org/wiki/Mebibyte#Multiple-byte_units Yeah we have long standing confusion that for historical reasons KB and MB often means 2^10 bytes units so now when you see KB you really don't know what it means. Therefore I am a staunch supporter of unambiguous KiB and MiB.
- gymbeaux 2y agoI think I explained this poorly, and I appear to be mistaken about it being to save CPU cycles (though an entity such as AWS would absolutely be about saving minuscule CPU cycles that add up at scale). I spent a lot of time researching this issue when I came across it years ago- I just remember that local file size and S3 file size was not matching up with anything and the takeaway was that S3 was calculating file size differently from Windows/Linux/macOS. AWS uses base 2 (binary) for calculating file size, meaning 1MB is treated as 1,048,576 (2^20) bytes rather than 1,000,000 bytes (10^6). And as you've said, one is MB while the other is technically MiB.
- gymbeaux 2y agoDownvotes notwithstanding https://repost.aws/knowledge-center/s3-console-metric-discrepancy https://repost.aws/knowledge-center/s3-console-metric-discre...