2 ms·
No there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POS
by aseipp 3y ago
No there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POSIX filesystem specification is notoriously vague about things like fsync consistency under particular scenarios, i.e. "do I need to fsync the parent directory before or after fsyncing the contents" for instance and has many bizarre performance cliffs if you aren't careful. At the scale AWS is at even a 10% performance cliff or performance delta would be worth clawing back if it meant removing the POSIX filesystem.
Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does.
> If you stand up an S3 instance with Ceph,
Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software).
See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.)
Many places do use POSIX filesystems for various reasons, even at large scale, of course.
[1] https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf