3 ms·

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy

Underneath the software, there’s still a filesystem with files. If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.

seabrookmx 3y ago
S3 actually uses a completely custom system[1] for writing bytes to disk. I haven't seen much in the way of details on the on-disk format but I certainly wouldn't assume it resembles a normal filesystem. [1]: https://aws.amazon.com/blogs/storage/how-automated-reasoning-helps-us-innovate-at-s3-scale/ https://aws.amazon.com/blogs/storage/how-automated-reasoning...
aseipp 3y ago
No there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POSIX filesystem specification is notoriously vague about things like fsync consistency under particular scenarios, i.e. "do I need to fsync the parent directory before or after fsyncing the contents" for instance and has many bizarre performance cliffs if you aren't careful. At the scale AWS is at even a 10% performance cliff or performance delta would be worth clawing back if it meant removing the POSIX filesystem. Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does. > If you stand up an S3 instance with Ceph, Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software). See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.) Many places do use POSIX filesystems for various reasons, even at large scale, of course. [1] https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf
yencabulator 3y ago
Ceph's BlueStore has talked direct to block devices, no filesystem in between, since 2017. https://ceph.com/community/new-luminous-bluestore/ https://ceph.com/community/new-luminous-bluestore/ [Disclaimer: ex-Ceph employee, from before BlueStore]
jandrewrogers 3y ago
I seriously doubt this is correct. It is common for database engines to install directly on raw block devices, bypassing the Linux kernel and effectively becoming the filesystem for those storage devices. Why would S3 work any differently? There are no advantages to building on top of a filesystem and many disadvantages for this kind of thing. It would be a poor engineering choice to build something like S3 on top of some other filesystem. There are often ways to do it by using an overlay that converts a filesystem into a pseudo block device, but that is usually considered a compatibility shim used for environments that lacking dedicated storage, at the cost of robustness and performance.

seabrookmx 3y ago

S3 actually uses a completely custom system[1] for writing bytes to disk. I haven't seen much in the way of details on the on-disk format but I certainly wouldn't assume it resembles a normal filesystem. [1]: https://aws.amazon.com/blogs/storage/how-automated-reasoning-helps-us-innovate-at-s3-scale/ https://aws.amazon.com/blogs/storage/how-automated-reasoning...

aseipp 3y ago

No there isn't. AWS does not use the traditional filesystem layer to store data; that would be a massive mistake from a performance and reliability POV; the POSIX filesystem specification is notoriously vague about things like fsync consistency under particular scenarios, i.e. "do I need to fsync the parent directory before or after fsyncing the contents" for instance and has many bizarre performance cliffs if you aren't careful. At the scale AWS is at even a 10% performance cliff or performance delta would be worth clawing back if it meant removing the POSIX filesystem. Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does. > If you stand up an S3 instance with Ceph, Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software). See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.) Many places do use POSIX filesystems for various reasons, even at large scale, of course. [1] https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

yencabulator 3y ago

Ceph's BlueStore has talked direct to block devices, no filesystem in between, since 2017. https://ceph.com/community/new-luminous-bluestore/ https://ceph.com/community/new-luminous-bluestore/ [Disclaimer: ex-Ceph employee, from before BlueStore]

jandrewrogers 3y ago

I seriously doubt this is correct. It is common for database engines to install directly on raw block devices, bypassing the Linux kernel and effectively becoming the filesystem for those storage devices. Why would S3 work any differently? There are no advantages to building on top of a filesystem and many disadvantages for this kind of thing. It would be a poor engineering choice to build something like S3 on top of some other filesystem. There are often ways to do it by using an overlay that converts a filesystem into a pseudo block device, but that is usually considered a compatibility shim used for environments that lacking dedicated storage, at the cost of robustness and performance.