7 ms·

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesystem, this can be mitigated somewhat by the disk cache, but for applications that would thrash the cache, cloud buckets are simply not a good storage backend if you desire disk-like access. What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts.

qsort 3y ago
My personal conspiracy theory: most "cloud services" are just... bad. VMs and disk space I understand completely, having machines on-prem is too much of an hassle and the price isn't that bad. But for stuff like this, managed services, databases especially, you're just getting scammed.
tyingq 3y ago
Well, and there's no such thing as opening a file and modifying some small part of it. That's emulated with a full rewrite of the whole object.
linsomniac 3y ago
>What I would really like to see is a two-tier cache system Is there any sort of Linux HSM (Hieracrhical Storage Manager)? I haven't see any and have been a bit surprised nothing has really developed there. They can manage putting hot data in RAM, SSDs, colder or larger data on spinning rust, deep freezing onto a tape silo or a cloud storage... Some of the NAS devices and RAID cards can support a two-tier caching or data migration using SSDs, where hot or highly-random data (usually identified by smaller write sizes) go to the SSDs, and then can migrate to the spinning discs. I've done some "poor mans" version of this using LVM, where I can "pvmove" blocks of a logical volume between spinning discs and SSDs, which is pretty slick, but a very crude tool.
- pdimitar 3y ago
  I don't know about a manager per se but `bcachefs` for Linux seems to do a good chunk of what you're after.
- folmar 3y ago
  CASTOR comes to mind for a start. Take a look a the CERN paper https://iopscience.iop.org/article/10.1088/1742-6596/331/5/052015 https://iopscience.iop.org/article/10.1088/1742-6596/331/5/0... as they have a large use case.
- tadfisher 3y ago
  Not a general kernel facility that I know of. I use nfscache every day though; my Steam data directory lives on NFS, and I set up nfscache with a 100GB LRU storage. This way I can avoid the "backup/restore" dance and have all my games installed, at the cost of waiting up to a few minutes to warm the cache for a new game.
ape4 3y ago
That page shows a `mkdir` is 3 json commands. I wonder if its that many HTTP requests.
8organicbits 3y ago
Same experience. I remember opening a .docx in Word and watching it hang or studder at different operations. I think you'd need very reliable and low latency networking for this to be anything but a painful to use toy. I'd be curious to see how it works running on EC2, especially with an S3 endpoint in the VPC. Although I still think you'd be better suited by using S3 as an object store, given the option to built it right.
pjc50 3y ago
> What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts This is really hard to get right if the origin cloud storage is anything other than immutable. Otherwise you're in for a world of cache invalidation and consistency pain. I've gradually come round to the other opinion: there should be devices that sit on the PCIe/NVMe bus and provide a blob storage API rather than a block one, and there should be an operating system blob API that is similar to but not identical to the filesystem one.
crazygringo 3y ago
This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend for. The latency doesn't matter in these cases for one-off things or scripts that already take seconds/minutes/hours anyways. So no this isn't niche or a toy. It's a fantastic production tool for a lot of different common uses. It's not for everything but nothing is. Use the right tool for the job.
- ninkendo 3y ago
  The problem is that such systems have a habit of growing in scope until they reach a point where you really do need the more optimal access patterns of using the real HTTP APIs, and the inefficiencies of emulating the full filesystem API will gradually start to bite you. Maybe you’re lucky enough that that won’t happen, but it’s important to understand it for the quick hack job it is, IMO.
  - lazide 3y ago
    In most situations that time is years, decades, or ‘never’. Which is fine. Not everyone or everything scales faster than bandwidth and/or CPU is.
  - deleted 3y ago
    [deleted]
- MontyCarloHall 3y ago
  Applications for which filesystem-like access is important (i.e. requiring lots of POSIX file I/O system calls, e.g. read(2)/write(2)/lseek(2)) but latency is unimportant seem pretty niche to me. If you don't need any of the POSIX syscalls, it's not that much more difficult to work with bucket URLs vs. file paths — the general format is the same, i.e. slash-delimited file/directory hierarchies.
VikingCoder 3y ago
Uh, how does it perform from a Google Compute Engine Virtual Machine? If it performs well there, I could imagine that being pretty useful.
- MontyCarloHall 3y ago
  That is exactly where I tested it, and the latency was still abysmally poor (~1 second per file operation). I don’t even want to know how bad the latency would be outside of a cloud VM.
thraxil 3y ago
Yep. I once inherited a system where the previous team had used GCSFuse to back the `/etc/letsencrypt` directory on a cluster of nginx webservers. It "worked" and may have been a reasonable approach at the time they built it, avoiding setting up a single "master" to handle HTTP-01 challenges (and it was before GCP's HTTPS LB could handle more than a handful of domains/certificates). The problem was that as the number of domains/certificates it handled increased, nginx startup or config reload time got slower and slower as it insists on stat-ing and reading every single file in that directory in the process. It got high enough that it started running into request throttling on the storage bucket. It's no fun when `nginx -s reload` takes two minutes and sometimes fails completely.
- netheril96 3y ago
  The most wrong part of that previous team is to store private keys unencrypted in the cloud, not the performance part.
  - thraxil 3y ago
    I mean... literally every VM running nginx or apache that I've ever seen has had the SSL certs just sitting on the filesystem in /etc/ssl or /etc/letsencrypt or similar... All of letsencrypt's documentation points people in that direction.
  - oittaa 3y ago
    My understanding is that everything is encrypted by default in GCP. Though you need to manually configure encryption keys if you want to prevent Google ever having access to your data.
    - FooBarWidget 3y ago
      This I don't understand. Even if you configure KMS, those are still keys stored on Google infra.
      - oittaa 3y ago
ArtWomb 3y ago
>>> every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation gcsfuse latency is ok as it embodies "infinite sync & persistence" ;)
markstos 3y ago
I once evaluated using s3fuse for managing about 36 million images. The old storage model was on a filesystem so it was supposed to make a smooth transition to the cloud. AWS Premium Support wisely advised me against it, not just because of latency but also because the abstraction makes /far/ more API calls then a native solution would. After a bit of testing to confirm, I switched to using native API calls. That code was easy to write and the performance was great. I've been wary of cloud FUSE adapters ever since.
- lazide 3y ago
  FUSE adapters in general are not for a product/production use in my experience. They’re great for one off convenience use, or basic admin scripts.
ashishbijlani 3y ago
I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready. 1. https://github.com/extfuse/extfuse https://github.com/extfuse/extfuse
- vamega 3y ago
  Is work on this continuing (or restarting)? I had heard of this a few years ago, but thought the project was shelved.
  - ashishbijlani 3y ago
    The project is active (just not merged in the kernel yet). Please DM me for questions.
yuliyp 3y ago
> most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache Isn't this how most servers run normally? (parts of) files which are accessed are in page cache, the rest is on "disk"
bitL 3y ago
Moreover, there is no SLA on those FUSE adapters so putting it into any part of production is too risky.
renewiltord 3y ago
Catfs is not super production (there are some small changes you need to make in inode handling), but you can do this. We have it on top of goofys. They both need a few changes to work under load but what we do is quite standard: 1. Goofys for S3 FUSE 2. Catfs for local disk caching 3. Linux caches in memory 4. Mmap file means processes share it 5. One device then exports this over the network to other machines, each of which have an application layer disk cache. 6. Machines are linked via 10 GigE (we use SFP+). Overall the goofys and catfs guy (kahing) wrote very performant software. Big fan.

qsort 3y ago

My personal conspiracy theory: most "cloud services" are just... bad. VMs and disk space I understand completely, having machines on-prem is too much of an hassle and the price isn't that bad. But for stuff like this, managed services, databases especially, you're just getting scammed.

tyingq 3y ago

Well, and there's no such thing as opening a file and modifying some small part of it. That's emulated with a full rewrite of the whole object.

linsomniac 3y ago

>What I would really like to see is a two-tier cache system Is there any sort of Linux HSM (Hieracrhical Storage Manager)? I haven't see any and have been a bit surprised nothing has really developed there. They can manage putting hot data in RAM, SSDs, colder or larger data on spinning rust, deep freezing onto a tape silo or a cloud storage... Some of the NAS devices and RAID cards can support a two-tier caching or data migration using SSDs, where hot or highly-random data (usually identified by smaller write sizes) go to the SSDs, and then can migrate to the spinning discs. I've done some "poor mans" version of this using LVM, where I can "pvmove" blocks of a logical volume between spinning discs and SSDs, which is pretty slick, but a very crude tool.

pdimitar 3y ago
I don't know about a manager per se but `bcachefs` for Linux seems to do a good chunk of what you're after.
folmar 3y ago
CASTOR comes to mind for a start. Take a look a the CERN paper https://iopscience.iop.org/article/10.1088/1742-6596/331/5/052015 https://iopscience.iop.org/article/10.1088/1742-6596/331/5/0... as they have a large use case.
tadfisher 3y ago
Not a general kernel facility that I know of. I use nfscache every day though; my Steam data directory lives on NFS, and I set up nfscache with a 100GB LRU storage. This way I can avoid the "backup/restore" dance and have all my games installed, at the cost of waiting up to a few minutes to warm the cache for a new game.

ape4 3y ago

That page shows a `mkdir` is 3 json commands. I wonder if its that many HTTP requests.

8organicbits 3y ago

Same experience. I remember opening a .docx in Word and watching it hang or studder at different operations. I think you'd need very reliable and low latency networking for this to be anything but a painful to use toy. I'd be curious to see how it works running on EC2, especially with an S3 endpoint in the VPC. Although I still think you'd be better suited by using S3 as an object store, given the option to built it right.

pjc50 3y ago

> What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts This is really hard to get right if the origin cloud storage is anything other than immutable. Otherwise you're in for a world of cache invalidation and consistency pain. I've gradually come round to the other opinion: there should be devices that sit on the PCIe/NVMe bus and provide a blob storage API rather than a block one, and there should be an operating system blob API that is similar to but not identical to the filesystem one.

crazygringo 3y ago

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend for. The latency doesn't matter in these cases for one-off things or scripts that already take seconds/minutes/hours anyways. So no this isn't niche or a toy. It's a fantastic production tool for a lot of different common uses. It's not for everything but nothing is. Use the right tool for the job.

ninkendo 3y ago
The problem is that such systems have a habit of growing in scope until they reach a point where you really do need the more optimal access patterns of using the real HTTP APIs, and the inefficiencies of emulating the full filesystem API will gradually start to bite you. Maybe you’re lucky enough that that won’t happen, but it’s important to understand it for the quick hack job it is, IMO.
- lazide 3y ago
  In most situations that time is years, decades, or ‘never’. Which is fine. Not everyone or everything scales faster than bandwidth and/or CPU is.
- deleted 3y ago
  [deleted]
MontyCarloHall 3y ago
Applications for which filesystem-like access is important (i.e. requiring lots of POSIX file I/O system calls, e.g. read(2)/write(2)/lseek(2)) but latency is unimportant seem pretty niche to me. If you don't need any of the POSIX syscalls, it's not that much more difficult to work with bucket URLs vs. file paths — the general format is the same, i.e. slash-delimited file/directory hierarchies.

VikingCoder 3y ago

Uh, how does it perform from a Google Compute Engine Virtual Machine? If it performs well there, I could imagine that being pretty useful.

MontyCarloHall 3y ago
That is exactly where I tested it, and the latency was still abysmally poor (~1 second per file operation). I don’t even want to know how bad the latency would be outside of a cloud VM.

thraxil 3y ago

Yep. I once inherited a system where the previous team had used GCSFuse to back the `/etc/letsencrypt` directory on a cluster of nginx webservers. It "worked" and may have been a reasonable approach at the time they built it, avoiding setting up a single "master" to handle HTTP-01 challenges (and it was before GCP's HTTPS LB could handle more than a handful of domains/certificates). The problem was that as the number of domains/certificates it handled increased, nginx startup or config reload time got slower and slower as it insists on stat-ing and reading every single file in that directory in the process. It got high enough that it started running into request throttling on the storage bucket. It's no fun when `nginx -s reload` takes two minutes and sometimes fails completely.

netheril96 3y ago
The most wrong part of that previous team is to store private keys unencrypted in the cloud, not the performance part.
- thraxil 3y ago
  I mean... literally every VM running nginx or apache that I've ever seen has had the SSL certs just sitting on the filesystem in /etc/ssl or /etc/letsencrypt or similar... All of letsencrypt's documentation points people in that direction.
- oittaa 3y ago
  My understanding is that everything is encrypted by default in GCP. Though you need to manually configure encryption keys if you want to prevent Google ever having access to your data.
  - FooBarWidget 3y ago
    This I don't understand. Even if you configure KMS, those are still keys stored on Google infra.
    - oittaa 3y ago

ArtWomb 3y ago

>>> every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation gcsfuse latency is ok as it embodies "infinite sync & persistence" ;)

markstos 3y ago

I once evaluated using s3fuse for managing about 36 million images. The old storage model was on a filesystem so it was supposed to make a smooth transition to the cloud. AWS Premium Support wisely advised me against it, not just because of latency but also because the abstraction makes /far/ more API calls then a native solution would. After a bit of testing to confirm, I switched to using native API calls. That code was easy to write and the performance was great. I've been wary of cloud FUSE adapters ever since.

lazide 3y ago
FUSE adapters in general are not for a product/production use in my experience. They’re great for one off convenience use, or basic admin scripts.

ashishbijlani 3y ago

I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready. 1. https://github.com/extfuse/extfuse https://github.com/extfuse/extfuse

vamega 3y ago
Is work on this continuing (or restarting)? I had heard of this a few years ago, but thought the project was shelved.
- ashishbijlani 3y ago
  The project is active (just not merged in the kernel yet). Please DM me for questions.

yuliyp 3y ago

> most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache Isn't this how most servers run normally? (parts of) files which are accessed are in page cache, the rest is on "disk"

bitL 3y ago

Moreover, there is no SLA on those FUSE adapters so putting it into any part of production is too risky.

renewiltord 3y ago

Catfs is not super production (there are some small changes you need to make in inode handling), but you can do this. We have it on top of goofys. They both need a few changes to work under load but what we do is quite standard: 1. Goofys for S3 FUSE 2. Catfs for local disk caching 3. Linux caches in memory 4. Mmap file means processes share it 5. One device then exports this over the network to other machines, each of which have an application layer disk cache. 6. Machines are linked via 10 GigE (we use SFP+). Overall the goofys and catfs guy (kahing) wrote very performant software. Big fan.