10 ms·

File systems unfit as distributed storage back ends: lessons from Ceph evolution

264 points by r4um 7y ago

wmitty 7y ago
The link to the paper in the article requires an ACM subscription. Here is a link to the version hosted by the authors: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf
- dooglius 7y ago
  I loaded it just fine from the link
ngrilly 7y ago
I read DigitalOcean is using Ceph for its object and block storage. Do they use BlueStore in production too?
- lathiat 7y ago
  Fairly sure at a Ceph Days recently they did a talk mentioning they’re on Jewel which would still be FileStore. BlueStore is really only reaching production deployments and maturity quite recently last year or so.
  - ngrilly 7y ago
    Thanks! This probably means we can expect some IOPS and latency improvements when they'll upgrade :)
    - kklimonda 7y ago
      DO has upgraded their Ceph cluster to luminous back in 2018 according to this blog post: https://blog.digitalocean.com/block-storage-volume-performance-burst/ https://blog.digitalocean.com/block-storage-volume-performan...
      - ngrilly 7y ago
        Thanks for the link. I missed that. It looks like they use BlueStore :)
        lathiat 7y ago
        My mistake I think I’m confusing them with another provider. Maybe OVH?
angrygoat 7y ago
I ran a ~ 0.5 PB Ceph cluster for a few years, on quite old spinning disk hardware (bought second-hand.) It was great: it just worked, coped very well with hardware failures, told the operator what was happening. An extremely solid, well-engineered system. My thanks to the Ceph team :)
- yankcrime 7y ago
  Yup, adding to this: For about four years we ran a similar size Ceph cluster that supported a burgeoning public cloud platform with users making extensive use of block and object. We did this on a range of hardware from creaky second-hand SuperMicro boxes to newer all-flash Quanta-based hardware. Through questionable hardware selection as well as standard operational challenges such as upgrades and scaling, Ceph never let us down. The only time we had a major problem turned out to be our fault. The creaky machines we were using at the start 'lost' half of their memory during a round of power failure testing. We didn't have monitoring in place to spot this, and unfortunately it manifest after we lost a node which triggered a significant rebalancing operation across the cluster. With several machines missing 50% of their RAM, this quickly descended into a horrendous disk thrashing exercise. Again, all credit due to Ceph - we were able to coax the cluster back into life with no data loss.
zzzcpan 7y ago
I don't think the conventional wisdom of building on top of filesystems exists. In distributed systems you always naturally gravitate towards using raw storage devices instead of filesystems, it becomes obvious very early on that filesystems suck too much and only create problems. And it's the same with all the embedded database libraries, you really want to write your own, because none of the existing ones were made to address performance and operational problems that arise even in small distributed systems. But at the same time early on you don't yet know most of the problems and don't want to invest time implementing something you don't yet understand well enough, so you end up building on top of filesystems and embedded databases and making plenty of poor choices and learning on your mistakes.
- notacoward 7y ago
  As I pointed out to the authors, this wheel has turned a couple of times. In the late 90s, many distributed filesystems (and most cluster filesystems) used raw disks and their own format. This was a burden, both for the developers who had to maintain an entire low-level I/O stack in addition to the distributed parts and to the users who had to learn new tools to deal with these "alien" disks in their system (which also limited deployment flexibility). Thus, when the current crop - e.g. Ceph, Gluster, PVFS2 - came around, they went toward a more local-FS-based approach. All of the issues mentioned in the paper were still real, but on the hardware of the time (both disks and networks) those weren't the bottlenecks anyway so the convenience was worth it. Now the tradeoffs have shifted again, and so have the solutions. Context: I've been an originator/maintainer for multiple projects in this space, and currently work on a storage system where we add space in bigger increments than Ceph's entire worldwide installed base (according to numbers in the paper).
  - PaulHoule 7y ago
    The filesystem as an abstraction has been getting long in the tooth for a long time -- it's taken a long time for the industry to recognize this, but that's the way it is with filesystems because a boring filesystem that never munches your data is preferable to most people to an interesting filesystem. Some evidence: * The proliferation of Linux filesystems such as ext4, XFS, ZFS, btrfs, reiserfs, JFS, JFFS, bcachefs, etc. If any of those filesystems were truly adequate there wouldn't have to be so many. * Microsoft's failure to replace NTFS with ReFS. (Interacts with storage spaces in such a way that it will never be reliable) * Microsoft's giving up on the old WSL and replacing it with a virtualized linux kernel because metadata operations on NTFS are terribly slow compared to Linux but people don't usually notice until they try to use an NTFS volume as if it was an ext4 volume. * The popularity of object stores, systems like Ceph, S3, etc. * Numerous filesystems promising transaction support and then backing away from it (btrfs as mentioned in the article, NTFS, exFAT, etc.) * Proliferation of APIs to access filesystems more efficiently. On one hand there are asyncio access methods for filesystems which aren't quite as solid as asyncio for networks, on the other hand there is mmap which can cause your thread to block not only when you make an I/O call but later when you access memory. * Recently some filesystem based APIs that address the real issues (e.g. pwrite, pread) but astonishingly late in the game.
tannhaeuser 7y ago
This isn't surprising, but I guess the results need to be put into perspective with the use case for distributed file systems and NFS eg. reasonable scaling for static asset serving with excellent modularity, in particular when paired with node-local caches. Of course Ceph etc. won't scale to Google Search and Facebook levels, but it's still damn practical if you're scaling out from a single HTTP server to a load-balanced cluster of those without having to bring in whole new I/O infrastructures. And they help with cloud vendor lock-in as well; for example you can use CephFS on DO, OVH, and other providers.
j-pb 7y ago
From the Blog post: > We looked at this issue earlier. Fundamentally the tension here is that copy-on-write semantics don’t fit with the emerging zone interface semantics. While the paper writes: > It is not surprising that attempts to modify production file systems, such as XFS and ext4, to work with the zone interface have so far been unsuccessful [19, 68], primarily because these are overwrite file systems, whereas the zone interface requires a copy-on-write approach to data management This seems to be a contradiction, and I'd side with the original paper.
- ZoomZoomZoom 7y ago
  Yes, found it strange too. The third quote block directly states that copy-on-write design is encouraged by the current hardware.
  - psds2 7y ago
    Zone interface is specifically good at log structured CoW is what it says. Later they talk about how they tried using a write ahead log but that it does not work well for managing metadata in a distributed filesystem.
ph2082 7y ago
> “For its next-generation backend, the Ceph community is exploring techniques that reduce the CPU consumption, such as minimizing data serialization-deserialization, and using the SeaStar framework with a shared-nothing model…“ Seastart HTTPD throughput as mentioned on their site Between 5 to 10 CPU, it can achieve 2,000,000 HTTP request/sec. Just Vow. But If you look at Http Performance data on below URL, running similar configuration on clouds (AWS etc.) looks costly though. http://seastar.io/http-performance/ http://seastar.io/http-performance/ I wonder what would be cost of achieving similar performance on Hadoop stack.
- lossolo 7y ago
  Just remember that besides beefy bare metal server, dedicated specialized network cards, they are also using DPDK, which is user space network stack.
  - ph2082 7y ago
    Ok, say that is how achieving high throughput. Does it limited to FreeBSD only or can be used on other *nix as well?
    - benlwalker 7y ago
      DPDK runs on Linux and FreeBSD officially. All of the momentum is on Linux though* *I was recently part of an effort to add FreeBSD testing to DPDK because SPDK's FreeBSD test agent kept failing when we updated DPDK. I'm a core maintainer for SPDK, which is basically DPDK for storage.
- ddorian43 7y ago
  Dumb http-response benchmarks can be achieved everywhere. Seastar excels on complex (cpu,disk,ram,network) scenarios (big disk-based db) with oltp/olap and keeping SLA low.
- aasasd 7y ago
  > it can achieve 2,000,000 HTTP request/sec Is that including some db operations? Because otherwise eeeeh: https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r18&hw=...
cfors 7y ago
Slightly off topic but I love Adrian Coyler’s blog. Since I never pursued graduate studies in CS I never really got into reading research papers but would love to start reading some on my commute to work. Does anyone have any recommendations for finding interesting papers? Do I need to buy subscriptions? Is there a list of “recommended” papers to read like we have with programming literature e.g. The Pragmatic Programmer?
- heinrichhartman 7y ago
  ACM subscription helps, since their "Digital Library" has pretty good coverage of the CS literature. This is something that your employer might want to sponsor. A strategy to find interesting papers that are worth reading as a novice in the field, is to pick any current paper that is interesting to you and look for heavily cited references. The following websites have citation data for papers: [1] SemanticScholar.com [2] GoogleScholar.com [3] citeseerx.ist.psu.edu
yankcrime 7y ago
For anyone looking for more information and benchmarks on the performance improvements in recent versions of Ceph (and with Bluestore in particular), here's a write-up that was done as part of testing for infrastructure to support the Human Brain Project: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-h...
- marmaduke 7y ago
  Unfortunately Julia was just a prototype system that they are going to turn off. KNL made it into one of their main production systems (Jureca) as a "booster" module, but they use GPFS on the main storage system (IIRC).
gwern 7y ago
The end-to-end principle strikes again: lowest common denominator abstractions like filesystems are often incorrect, inefficient, or both for complex applications and ultimately must be bypassed by custom abstractions tailored for the application.
- anfilt 7y ago
  It's kinda why I think something like an exokernel would make more sense than way OSes trying to abstract things to a certain level. We should be trying to build abstractions that can be peeled layer by layer like an onion, not a potato.
  - kragen 7y ago
    Unix files aren't quite an exokernel-style "just securely multiplex the disk" but in some ways they come closer than some other alternatives. No file types, just randomly accessible bytes (well, and an execute bit); no multiple streams; no ISAM, just bytes (except that you do have directories); no insertion (but you do have append and truncate). You could make them more disk-like by making them fixed-size, with the size specified at creation time, and accessing them in blocks rather than in bytes. Would those be an improvement? I tend to think not. Certainly if those were the semantics provided by the kernel you would want userland filesystem processes to provide appendable files. Copy-on-write file versioning and cross-file, cross-process transactions, on the other hand, could be real pluses. I'd be okay with those being provided by userland processes rather than the kernel, but I'd sure like to have them.
rsync 7y ago
I have sympathy with, and am open-minded to, the conclusions of this article - even as a die-hard, true believer in the filesystem (esp. ZFS) as a useful foundational building block. However, I hope that these conclusions do not lead to the intentional deprecation of support for filesystems in projects like Ceph. If a non-filesystem backing store is superior, then by all means do it, but I hope the ability to deploy a filesystem-backed endpoint will be retained. In a pinch, it's very flexible and there are a lot of them lying around ...
- amluto 7y ago
  If nothing else, most filesystems can expose a big file that can be used, with decent efficiency, as though it was a disk. For COW filesystems, turning off COW on the file may improve performance, and, if the FS is on RAID, the resulting performance and correctness properties will be odd.
  - pm7 7y ago
    But if services will need to use one big file, we won't have possibility to overcommit disk space (unless they implement something like TRIM).
newnewpdro 7y ago
Don't most of their problems go away when you fallocate a pile of space and use AIO+O_DIRECT like a database to get the buffer cache and most of the filesystem out of the way? CoW filesystems like BTRFS proviode ioctls to disable CoW as well, which would be useful here when you've grown your own. XFS has supported shutting off metadata like ctime/mtime updates for ages. If you jump through some hoops, with a fully allocated file, you can get a file on a filesystem to behave very closely to a bare block store.
- notacoward 7y ago
  > Don't most of their problems go away No. Yes, you can fallocate. Yes, you can use AIO, or even better io_uring. Yes, you can use O_DIRECT ... if you want to give up caching and have to deal with alignment restrictions, and it turns out that O_DIRECT turns into O_SYNC in some unexpected edge cases. That gets you something that kind of sort of behaves like a block device, but that doesn't solve any of the problems the authors identify. * No transactional semantics. Roll your own. (I'm actually OK with this one BTW, but others feel differently.) * No help with slow metadata operations. Still slow if you're still using multiple files, or roll your own within a single file. * No improvement in support for new media types (e.g. shingled/zoned). In an ideal world, local filesystems would do a decent job supporting distributed filesystems (and other data stores). Instead, we're in a world where local filesystems fall short in many ways, and the solution to every deficiency is to avoid the local filesystem as much as possible. That way leads to silos and lock-in, so I don't think it's a good answer. Local filesystems need to be better, or someone needs to create an equally standard abstraction and set of tools to do what local filesytems can't.
gambler 7y ago
I was just discussing with a colleague how technology accretes and how no one reevaluates high-level design decisions even after every single factor leading to those decisions has changed. It's weird that basic filesystems today are so out of touch with modern realities that we are universally forced to resort to using complex databases even in cases when the logical model of files and directories fits the storage needs really well. It's weird that hierarchical storage is the only universal model available on all OSes and in all languages. The more I think about it, the more I realize that we live in a bizarro world where software runs everything, yet makes little to no sense from either human, or modern hardware or system design perspectives.
- kragen 7y ago
  I'm interested to hear what kinds of filesystem semantics you think might strike a better balance!
- nostrademons 7y ago
  Programming languages are in the same boat - the modern CPU works dramatically different from a PDP-11, but most programming models still assume an accumulator machine; flat memory hierarchy; effectively unbounded LIFO program-control stacks; uniform machine word sizes; sequential in-order execution; and byte streams for I/O. This is despite large register files, multiple levels of caching, coroutines/promises, SIMD, multicore/GPU, and page-level I/O being things. In many cases even assembly language encodes assumptions from the 1970s, and then is internally translated by the processor into how the hardware actually works.
  - adfkbuvasdas 7y ago
    It’s not a totally fair analogy though. Designing file systems for newer storage devices is an engineering problem that requires a lot of effort, but not a great amount of new insight. Designing a new programming language that somehow exposes the concepts you mentioned more transparently while still being actually useful would require major, major breakthroughs and insights in PL design.
  - h0l0cube 7y ago
throw0101a 7y ago
We're actually facing an issue with our Ceph infrastructure in the 'upgrade' from FileStore to BlueStore: the loss of use of our SSDs. We created our infrastructure with a bunch of hardware that had HDDs for bulk storage and an SSD for async I/O and intent log stuff. The problem is that BlueStore does not seem to have any use for off-to-the-side SSDs AFA(we)CT. So we're left with a bunch hardware that may not be as performant under the new BlueStore world order. The Ceph mailing list consensus seems to be "don't buy SSDs, but rather buy more spindles for more independent OSDs". That's fine for future purchases, but we have a whole bunch of gear designed for the Old Way of doing things. We could leave things be and continue using FileStore, but it seems the Path Forward is BlueStore. Some of us do not need the speed of an all-SSD setup, but perhaps want something a little faster than only-HDDs. We're playing with benchmarks now to see how much worse the latency is with BlueStore+no-SSD, and whether the latency is good enough for us as-is. Any new storage design that cannot handle an "hybrid" configuration of combining HDDs and SSDs is silly IMHO. I joked that we could tie the HDDs together using ZFS zvol, with the SSD as the ZIL, and point the OSD(s) there.
- Nullabillity 7y ago
  From what I can tell, BlueStore still supports using separate disks for the WAL/RocksDB[0]? [0]: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#devices https://docs.ceph.com/docs/master/rados/configuration/bluest...
  - throw0101a 7y ago
    Yes, the confounding factor was/is we are using ceph-ansible with ceph-disk. If we want to upgrade we have to make a whole bunch of inter-related changes. (I'm not the lead on the project, so no doubt have forgotten some of the exact complications involved. Though it does seem slight strange (IMHO) that you're getting rid of the file system, but still keep the LVM layer.)
Mathnerd314 7y ago
I guess a minimal install of Ceph is a client and a node with 2-3 hard disks / SSDs.
bullen 7y ago
I use ext4 with my distributed async-to-async JSON database: https://github.com/tinspin/rupy/wiki/Storage https://github.com/tinspin/rupy/wiki/Storage You can try it here: http://root.rupy.se http://root.rupy.se The actual syncing is done over HTTP with Java though so maybe that's why it works well for me.
lightedman 7y ago
Unfit? That's why I'm using KirbyCMS atop NTFS with zero issues, right?
- lightedman 7y ago
  https://news.ycombinator.com/item?id=13679424 https://news.ycombinator.com/item?id=13679424 Older thread with people having discussed this that back up my words right up top. I don't have a single issue with KirbyCMS + NTFS, I have distributed back end stuff as I desire, and it just works and has mature documentation.

264 points by r4um 7y ago

The link to the paper in the article requires an ACM subscription. Here is a link to the version hosted by the authors: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

dooglius 7y ago
I loaded it just fine from the link

ngrilly 7y ago

I read DigitalOcean is using Ceph for its object and block storage. Do they use BlueStore in production too?

lathiat 7y ago
Fairly sure at a Ceph Days recently they did a talk mentioning they’re on Jewel which would still be FileStore. BlueStore is really only reaching production deployments and maturity quite recently last year or so.
- ngrilly 7y ago
  Thanks! This probably means we can expect some IOPS and latency improvements when they'll upgrade :)
  - kklimonda 7y ago
    DO has upgraded their Ceph cluster to luminous back in 2018 according to this blog post: https://blog.digitalocean.com/block-storage-volume-performance-burst/ https://blog.digitalocean.com/block-storage-volume-performan...
    - ngrilly 7y ago
      Thanks for the link. I missed that. It looks like they use BlueStore :)
      - lathiat 7y ago
        My mistake I think I’m confusing them with another provider. Maybe OVH?

angrygoat 7y ago

I ran a ~ 0.5 PB Ceph cluster for a few years, on quite old spinning disk hardware (bought second-hand.) It was great: it just worked, coped very well with hardware failures, told the operator what was happening. An extremely solid, well-engineered system. My thanks to the Ceph team :)

yankcrime 7y ago
Yup, adding to this: For about four years we ran a similar size Ceph cluster that supported a burgeoning public cloud platform with users making extensive use of block and object. We did this on a range of hardware from creaky second-hand SuperMicro boxes to newer all-flash Quanta-based hardware. Through questionable hardware selection as well as standard operational challenges such as upgrades and scaling, Ceph never let us down. The only time we had a major problem turned out to be our fault. The creaky machines we were using at the start 'lost' half of their memory during a round of power failure testing. We didn't have monitoring in place to spot this, and unfortunately it manifest after we lost a node which triggered a significant rebalancing operation across the cluster. With several machines missing 50% of their RAM, this quickly descended into a horrendous disk thrashing exercise. Again, all credit due to Ceph - we were able to coax the cluster back into life with no data loss.

zzzcpan 7y ago

I don't think the conventional wisdom of building on top of filesystems exists. In distributed systems you always naturally gravitate towards using raw storage devices instead of filesystems, it becomes obvious very early on that filesystems suck too much and only create problems. And it's the same with all the embedded database libraries, you really want to write your own, because none of the existing ones were made to address performance and operational problems that arise even in small distributed systems. But at the same time early on you don't yet know most of the problems and don't want to invest time implementing something you don't yet understand well enough, so you end up building on top of filesystems and embedded databases and making plenty of poor choices and learning on your mistakes.

notacoward 7y ago
As I pointed out to the authors, this wheel has turned a couple of times. In the late 90s, many distributed filesystems (and most cluster filesystems) used raw disks and their own format. This was a burden, both for the developers who had to maintain an entire low-level I/O stack in addition to the distributed parts and to the users who had to learn new tools to deal with these "alien" disks in their system (which also limited deployment flexibility). Thus, when the current crop - e.g. Ceph, Gluster, PVFS2 - came around, they went toward a more local-FS-based approach. All of the issues mentioned in the paper were still real, but on the hardware of the time (both disks and networks) those weren't the bottlenecks anyway so the convenience was worth it. Now the tradeoffs have shifted again, and so have the solutions. Context: I've been an originator/maintainer for multiple projects in this space, and currently work on a storage system where we add space in bigger increments than Ceph's entire worldwide installed base (according to numbers in the paper).
- PaulHoule 7y ago
  The filesystem as an abstraction has been getting long in the tooth for a long time -- it's taken a long time for the industry to recognize this, but that's the way it is with filesystems because a boring filesystem that never munches your data is preferable to most people to an interesting filesystem. Some evidence: * The proliferation of Linux filesystems such as ext4, XFS, ZFS, btrfs, reiserfs, JFS, JFFS, bcachefs, etc. If any of those filesystems were truly adequate there wouldn't have to be so many. * Microsoft's failure to replace NTFS with ReFS. (Interacts with storage spaces in such a way that it will never be reliable) * Microsoft's giving up on the old WSL and replacing it with a virtualized linux kernel because metadata operations on NTFS are terribly slow compared to Linux but people don't usually notice until they try to use an NTFS volume as if it was an ext4 volume. * The popularity of object stores, systems like Ceph, S3, etc. * Numerous filesystems promising transaction support and then backing away from it (btrfs as mentioned in the article, NTFS, exFAT, etc.) * Proliferation of APIs to access filesystems more efficiently. On one hand there are asyncio access methods for filesystems which aren't quite as solid as asyncio for networks, on the other hand there is mmap which can cause your thread to block not only when you make an I/O call but later when you access memory. * Recently some filesystem based APIs that address the real issues (e.g. pwrite, pread) but astonishingly late in the game.

tannhaeuser 7y ago

This isn't surprising, but I guess the results need to be put into perspective with the use case for distributed file systems and NFS eg. reasonable scaling for static asset serving with excellent modularity, in particular when paired with node-local caches. Of course Ceph etc. won't scale to Google Search and Facebook levels, but it's still damn practical if you're scaling out from a single HTTP server to a load-balanced cluster of those without having to bring in whole new I/O infrastructures. And they help with cloud vendor lock-in as well; for example you can use CephFS on DO, OVH, and other providers.

j-pb 7y ago

From the Blog post: > We looked at this issue earlier. Fundamentally the tension here is that copy-on-write semantics don’t fit with the emerging zone interface semantics. While the paper writes: > It is not surprising that attempts to modify production file systems, such as XFS and ext4, to work with the zone interface have so far been unsuccessful [19, 68], primarily because these are overwrite file systems, whereas the zone interface requires a copy-on-write approach to data management This seems to be a contradiction, and I'd side with the original paper.

ZoomZoomZoom 7y ago
Yes, found it strange too. The third quote block directly states that copy-on-write design is encouraged by the current hardware.
- psds2 7y ago
  Zone interface is specifically good at log structured CoW is what it says. Later they talk about how they tried using a write ahead log but that it does not work well for managing metadata in a distributed filesystem.

ph2082 7y ago

> “For its next-generation backend, the Ceph community is exploring techniques that reduce the CPU consumption, such as minimizing data serialization-deserialization, and using the SeaStar framework with a shared-nothing model…“ Seastart HTTPD throughput as mentioned on their site Between 5 to 10 CPU, it can achieve 2,000,000 HTTP request/sec. Just Vow. But If you look at Http Performance data on below URL, running similar configuration on clouds (AWS etc.) looks costly though. http://seastar.io/http-performance/ http://seastar.io/http-performance/ I wonder what would be cost of achieving similar performance on Hadoop stack.

lossolo 7y ago
Just remember that besides beefy bare metal server, dedicated specialized network cards, they are also using DPDK, which is user space network stack.
- ph2082 7y ago
  Ok, say that is how achieving high throughput. Does it limited to FreeBSD only or can be used on other *nix as well?
  - benlwalker 7y ago
    DPDK runs on Linux and FreeBSD officially. All of the momentum is on Linux though* *I was recently part of an effort to add FreeBSD testing to DPDK because SPDK's FreeBSD test agent kept failing when we updated DPDK. I'm a core maintainer for SPDK, which is basically DPDK for storage.
ddorian43 7y ago
Dumb http-response benchmarks can be achieved everywhere. Seastar excels on complex (cpu,disk,ram,network) scenarios (big disk-based db) with oltp/olap and keeping SLA low.
aasasd 7y ago
> it can achieve 2,000,000 HTTP request/sec Is that including some db operations? Because otherwise eeeeh: https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r18&hw=...

cfors 7y ago

Slightly off topic but I love Adrian Coyler’s blog. Since I never pursued graduate studies in CS I never really got into reading research papers but would love to start reading some on my commute to work. Does anyone have any recommendations for finding interesting papers? Do I need to buy subscriptions? Is there a list of “recommended” papers to read like we have with programming literature e.g. The Pragmatic Programmer?

heinrichhartman 7y ago
ACM subscription helps, since their "Digital Library" has pretty good coverage of the CS literature. This is something that your employer might want to sponsor. A strategy to find interesting papers that are worth reading as a novice in the field, is to pick any current paper that is interesting to you and look for heavily cited references. The following websites have citation data for papers: [1] SemanticScholar.com [2] GoogleScholar.com [3] citeseerx.ist.psu.edu

yankcrime 7y ago

For anyone looking for more information and benchmarks on the performance improvements in recent versions of Ceph (and with Bluestore in particular), here's a write-up that was done as part of testing for infrastructure to support the Human Brain Project: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-h...

marmaduke 7y ago
Unfortunately Julia was just a prototype system that they are going to turn off. KNL made it into one of their main production systems (Jureca) as a "booster" module, but they use GPFS on the main storage system (IIRC).

gwern 7y ago

The end-to-end principle strikes again: lowest common denominator abstractions like filesystems are often incorrect, inefficient, or both for complex applications and ultimately must be bypassed by custom abstractions tailored for the application.

anfilt 7y ago
It's kinda why I think something like an exokernel would make more sense than way OSes trying to abstract things to a certain level. We should be trying to build abstractions that can be peeled layer by layer like an onion, not a potato.
- kragen 7y ago
  Unix files aren't quite an exokernel-style "just securely multiplex the disk" but in some ways they come closer than some other alternatives. No file types, just randomly accessible bytes (well, and an execute bit); no multiple streams; no ISAM, just bytes (except that you do have directories); no insertion (but you do have append and truncate). You could make them more disk-like by making them fixed-size, with the size specified at creation time, and accessing them in blocks rather than in bytes. Would those be an improvement? I tend to think not. Certainly if those were the semantics provided by the kernel you would want userland filesystem processes to provide appendable files. Copy-on-write file versioning and cross-file, cross-process transactions, on the other hand, could be real pluses. I'd be okay with those being provided by userland processes rather than the kernel, but I'd sure like to have them.

rsync 7y ago

I have sympathy with, and am open-minded to, the conclusions of this article - even as a die-hard, true believer in the filesystem (esp. ZFS) as a useful foundational building block. However, I hope that these conclusions do not lead to the intentional deprecation of support for filesystems in projects like Ceph. If a non-filesystem backing store is superior, then by all means do it, but I hope the ability to deploy a filesystem-backed endpoint will be retained. In a pinch, it's very flexible and there are a lot of them lying around ...

amluto 7y ago
If nothing else, most filesystems can expose a big file that can be used, with decent efficiency, as though it was a disk. For COW filesystems, turning off COW on the file may improve performance, and, if the FS is on RAID, the resulting performance and correctness properties will be odd.
- pm7 7y ago
  But if services will need to use one big file, we won't have possibility to overcommit disk space (unless they implement something like TRIM).

newnewpdro 7y ago

Don't most of their problems go away when you fallocate a pile of space and use AIO+O_DIRECT like a database to get the buffer cache and most of the filesystem out of the way? CoW filesystems like BTRFS proviode ioctls to disable CoW as well, which would be useful here when you've grown your own. XFS has supported shutting off metadata like ctime/mtime updates for ages. If you jump through some hoops, with a fully allocated file, you can get a file on a filesystem to behave very closely to a bare block store.

notacoward 7y ago
> Don't most of their problems go away No. Yes, you can fallocate. Yes, you can use AIO, or even better io_uring. Yes, you can use O_DIRECT ... if you want to give up caching and have to deal with alignment restrictions, and it turns out that O_DIRECT turns into O_SYNC in some unexpected edge cases. That gets you something that kind of sort of behaves like a block device, but that doesn't solve any of the problems the authors identify. * No transactional semantics. Roll your own. (I'm actually OK with this one BTW, but others feel differently.) * No help with slow metadata operations. Still slow if you're still using multiple files, or roll your own within a single file. * No improvement in support for new media types (e.g. shingled/zoned). In an ideal world, local filesystems would do a decent job supporting distributed filesystems (and other data stores). Instead, we're in a world where local filesystems fall short in many ways, and the solution to every deficiency is to avoid the local filesystem as much as possible. That way leads to silos and lock-in, so I don't think it's a good answer. Local filesystems need to be better, or someone needs to create an equally standard abstraction and set of tools to do what local filesytems can't.

gambler 7y ago

I was just discussing with a colleague how technology accretes and how no one reevaluates high-level design decisions even after every single factor leading to those decisions has changed. It's weird that basic filesystems today are so out of touch with modern realities that we are universally forced to resort to using complex databases even in cases when the logical model of files and directories fits the storage needs really well. It's weird that hierarchical storage is the only universal model available on all OSes and in all languages. The more I think about it, the more I realize that we live in a bizarro world where software runs everything, yet makes little to no sense from either human, or modern hardware or system design perspectives.

kragen 7y ago
I'm interested to hear what kinds of filesystem semantics you think might strike a better balance!
nostrademons 7y ago
Programming languages are in the same boat - the modern CPU works dramatically different from a PDP-11, but most programming models still assume an accumulator machine; flat memory hierarchy; effectively unbounded LIFO program-control stacks; uniform machine word sizes; sequential in-order execution; and byte streams for I/O. This is despite large register files, multiple levels of caching, coroutines/promises, SIMD, multicore/GPU, and page-level I/O being things. In many cases even assembly language encodes assumptions from the 1970s, and then is internally translated by the processor into how the hardware actually works.
- adfkbuvasdas 7y ago
  It’s not a totally fair analogy though. Designing file systems for newer storage devices is an engineering problem that requires a lot of effort, but not a great amount of new insight. Designing a new programming language that somehow exposes the concepts you mentioned more transparently while still being actually useful would require major, major breakthroughs and insights in PL design.
- h0l0cube 7y ago

throw0101a 7y ago

We're actually facing an issue with our Ceph infrastructure in the 'upgrade' from FileStore to BlueStore: the loss of use of our SSDs. We created our infrastructure with a bunch of hardware that had HDDs for bulk storage and an SSD for async I/O and intent log stuff. The problem is that BlueStore does not seem to have any use for off-to-the-side SSDs AFA(we)CT. So we're left with a bunch hardware that may not be as performant under the new BlueStore world order. The Ceph mailing list consensus seems to be "don't buy SSDs, but rather buy more spindles for more independent OSDs". That's fine for future purchases, but we have a whole bunch of gear designed for the Old Way of doing things. We could leave things be and continue using FileStore, but it seems the Path Forward is BlueStore. Some of us do not need the speed of an all-SSD setup, but perhaps want something a little faster than only-HDDs. We're playing with benchmarks now to see how much worse the latency is with BlueStore+no-SSD, and whether the latency is good enough for us as-is. Any new storage design that cannot handle an "hybrid" configuration of combining HDDs and SSDs is silly IMHO. I joked that we could tie the HDDs together using ZFS zvol, with the SSD as the ZIL, and point the OSD(s) there.

Nullabillity 7y ago
From what I can tell, BlueStore still supports using separate disks for the WAL/RocksDB[0]? [0]: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#devices https://docs.ceph.com/docs/master/rados/configuration/bluest...
- throw0101a 7y ago
  Yes, the confounding factor was/is we are using ceph-ansible with ceph-disk. If we want to upgrade we have to make a whole bunch of inter-related changes. (I'm not the lead on the project, so no doubt have forgotten some of the exact complications involved. Though it does seem slight strange (IMHO) that you're getting rid of the file system, but still keep the LVM layer.)

Mathnerd314 7y ago

I guess a minimal install of Ceph is a client and a node with 2-3 hard disks / SSDs.

bullen 7y ago

I use ext4 with my distributed async-to-async JSON database: https://github.com/tinspin/rupy/wiki/Storage https://github.com/tinspin/rupy/wiki/Storage You can try it here: http://root.rupy.se http://root.rupy.se The actual syncing is done over HTTP with Java though so maybe that's why it works well for me.

lightedman 7y ago

Unfit? That's why I'm using KirbyCMS atop NTFS with zero issues, right?

lightedman 7y ago
https://news.ycombinator.com/item?id=13679424 https://news.ycombinator.com/item?id=13679424 Older thread with people having discussed this that back up my words right up top. I don't have a single issue with KirbyCMS + NTFS, I have distributed back end stuff as I desire, and it just works and has mature documentation.