10 ms·

Google and Red Hat announce cloud-based scalable file servers

227 points by Sami_Lehtinen 11y ago

prohor 11y ago
I'm not sure what is announced here. Gluster FS is for few years already (version 3.6 now), while the article doesn't mention that there is started any managed service based on it. It is more like reminder that you can set up distributed file system on your cloud servers using Gluster. Not even any step-by-step tutorial how to do that.
- baldfat 11y ago
  > Google Cloud Platform and Red Hat are proud to announce the availability of Red Hat Gluster Storage on Google Compute Engine. Gluster is available not announcing that this is a new technology.
  - prohor 11y ago
    How this availability will be implemented then? I can install it now, so it is already available for me. Will they provide a service like Amazon Elastic File System? Or there will be some pre-configured images?
    - bonzini 11y ago
      On AWS you get pre-configured AMIs, I suppose it's similar.
  - ssmoot 11y ago
    Not sure what you're saying?
  - profmonocle 11y ago
    That's what I find confusing. Was it not available before now? Couldn't you just spin up some Red Hat instances and run it? It's not clear what's actually being announced - it doesn't sound like a managed service.
- dragonwriter 11y ago
  > I'm not sure what is announced here. I think, though I'm not sure, that its an official, active support relationship between Google and Red Hat for GlusterFS on GCE, related to Red Hat's vendor certification programs. If so, this seems like the kind of thing that has the most impact on people with support contracts with Red Hat, but should also have peripheral impacts on the stability, quality of support, etc., of the product on the GCE platform generally.
- milesward 11y ago
  Disclaimer: I work on GCP. Fixed that for you, check out this: https://cloud.google.com/solutions/filers-on-compute-engine https://cloud.google.com/solutions/filers-on-compute-engine
secopdev 11y ago
pricing?
- milesward 11y ago
  Disclaimer: I work on GCP. GlusterFS works best on RHEL, and consumes normal GCP resources like GCE and PD-SSD storage. To host a rocking fast, best practices, HA, 3TB all-SSD filer, it'd be less than $900 on GCP: https://cloud.google.com/products/calculator/#id=e76e9a5a-bf21-4ccd-9cab-5590c77394db https://cloud.google.com/products/calculator/#id=e76e9a5a-bf...
jqueryin 11y ago
Before reading the article, I was going to ask if it solves the "high read access of many small files" I/O problem, but alas, it's on GlusterFS, so only insomuch as Gluster has been making improvements these last few minor releases. Is anyone here running a GlusterFS setup with high read/write volume on small files successfully? If so, what's your secret?
cgarrigue 11y ago
It's a bit light for a press release. Considering RedHat is officially promoting AWS on their website, providing more information to let people know whether the offering on Google Cloud will be better or similar would have been better.
- justinclift 11y ago
  Yeah, it seems to be missing the "click here for next steps / further information" bit, which is less than optimal. :(
  - milesward 11y ago
    Disclaimer: I work on GCP Yup, we goofed. Here's the next step bits: https://cloud.google.com/solutions/filers-on-compute-engine https://cloud.google.com/solutions/filers-on-compute-engine
    - justinclift 11y ago
      Ugh. So it's literally still RH Gluster, needing manual setup. Also, there's a sentence on the end of the GlusterFS section there saying: "If you want to deploy a Red Hat Gluster Storage cluster on Compute Engine, see this white paper for instructions on how to provision a multi-node cluster that includes cross-zone and cross-region replication:" Apart from the typo (pedant alert!), the URL on "white paper" goes to a non-public document only Red Hat subscribers have access to. That should probably be that fixed, so non-Red-Hat-subscribers can read the doc and know what they'll need to do up front. If people need to subscribe to RH in order to get that info... just to know what they need to do... that's probably going to hinder adoption. Potentially by a lot. ;)
pilif 11y ago
Last time I tried GlusterFS was in 2012. The way it worked was very impressive back then and I would have loved to actually put it into production. Unfortunately, I hit a roadblock in relation to enumeration of huge directories: Even with just 5K files in a directory, performance started to drop really badly to the point where enumerating a directory containing 10K files would take longer than 5 minutes. Yes. You're not supposed to store many files in a directory, but this was about giving third parties FTP upload access for product pictures and I can't possibly ask them to follow any schema for file and folder naming. These people want a directory to put stuff to with their GUI FTP client and they want their client to be able to not upload files if the target already exists. So having all files in one directory was a huge improvement UX-wise. So in the end, I had to move to nfs on top of drbd to provide shared backend storage. Enumerating 20K files over NFS still isn't fast but completes within 2 seconds instead of more than 5 minutes. Of course, now that we're talking about GlusterFS, I wonder whether this has been fixed since?
- mikaelj 11y ago
  Couldn't your FTP server have handled this in a clever way? For example, sort in directories by first letter, then by first two letters. While still providing a virtual flat view to the FTP client user. It'd be a simple mapping away.
  - pilif 11y ago
    You assume I wanted to write an FTP server. I don't. I am using stock vsftp with a PAM module that allows authentication against our web application.
- jweir 11y ago
  I started using Gluster this last fall. At first it was not a contender - the articles I read were not encouraging. However many of this articles were older and it appears a lot of progress has been made with Gluster since version 3.5 It maybe worth a second look - however a directory with a large file count still might be a problem
justinclift 11y ago
Heyas, Ex-GlusterFS person here (used to work at Red Hat on the project side, leaving mid last year). "Small file access", and "lots of files in a directory" have been a pain point with GlusterFS for ages. The 3.7.0 release had some important improvements in it, specificially designed to help fix that: https://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf https://www.gluster.org/community/documentation/index.php/Fe... The latest Gluster release is 3.7.8 (the same series as 3.7.0), and is worth looking at if you're needing a good distributed file system. If you have something like 1Mill files in a single directory though... hrmmm... NFS or other technologies might still be a better idea. ;)
- DannoHung 11y ago
  What makes having a lot of files in a directory a hard problem?
  - justinclift 11y ago
    From memory, it was something about not having a central metadata server combined with needing to talk with each of the nodes to get their idea of "the latest version" of file info. Especially tricky when taking into account edge cases, such as when some nodes are failed/down/etc. Unfortunately, I don't remember most of the details. It's fallen out of my headspace in the months since I've left. ;)
  - TallGuyShort 11y ago
    It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does both extremely well is non-trivial. You can optimize to store a few very large files or many very small files, but doing both while scaling reliably is harder than you might think.
    - DannoHung 11y ago
      Naively it seems weird since metadata is just data, right? So if you can store a large amount of data in a file, you should be able to store the metadata But I don't know shit about filesystem design, soooooo...
godzillabrennus 11y ago
Glad to see Gluster is still making waves. I was an early customer. It's impressive when a brand survives acquisition much less a transition into a new type of offering like this. Kudos to everyone who helped make Gluster special!
thrownaway2424 11y ago
I see that the GlusterFS FAQ says it is fully POSIX compliant. That's a pretty good trick. Ten years ago or so I had a suite of compliance tests I would use to embarrass salesmen from iBrix and Panasas. The only actually POSIX-compliant distributed filesystem I could find in those days was Lustre (unrelated to Gluster, despite the naming). Lustre works well but it almost impossible to install and operate.
- kbenson 11y ago
  I remember at around the same time I was playing with GlusterFS, and it was interesting, but painful to configure as well. It was prone to have client/server configuration mismatches, and you had to be careful to reason about where you enabled a feature, the client or server, to make sure it failed in a safe way. It did have capabilities for Fcntl and flock file locking though, which was interesting at the time. Unfortunately ist was also somewhat unstable, and I would see segfaults every few weeks. My focus for work projects moved into different areas, and I didn't keep up with it's development, so I'm not sure what they've done in the last decade, but it was promising and refreshingly new back then. I should take another look.
- batbomb 11y ago
  In HPC/HTC, Lustre is very common, especially in DOE labs. I've never tried to install it, but I don't think my colleagues who have are some kind of special genius.
  - jpgvm 11y ago
    Most HPC labs run distributions that go to quite a lot of pain to ensure Lustre works well. The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues. That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install. Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park. Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.
chatmasta 11y ago
I needed a shared volume across multiple EC2 instances in a VPC. My use case is that multiple "ingress" boxes write files to the shared volume, and then a single "worker" box processes those files. This is a somewhat unusual use case in that it means one box is responsible for 99% of IO heavy operations, and the other boxes are responsible only for writing to the volume, with no latency requirements. My solution was to mount an EBS on the "worker box," along with an NFS server. Each "ingress box" runs an NFS client that connects to the server via its internal VPC IP address, and mounts the NFS volume to a local directory. It works wonderfully. In three months of running this setup, I've had no downtime or issues, not even minor ones. Granted I don't need any kind of extreme I/O performance, so I haven't measured it, but this system took less than an hour to setup and fit my needs perfectly.
- helper 11y ago
  Amazon now has an NFS service (in preview release) called EFS: https://aws.amazon.com/efs/ https://aws.amazon.com/efs/
deleted 11y ago
[deleted]
Beldur 11y ago
Red Hat announcement: http://www.redhat.com/en/about/press-releases/red-hat-unveils-flexible-and-portable-cloud-storage-red-hat-gluster-storage-google-cloud-platform http://www.redhat.com/en/about/press-releases/red-hat-unveil...
gamegod 11y ago
I worked with a GlusterFS deployment in production about 2 years ago, and it was such a nightmare that I both feel compelled to write about it and never touch anything made by that team ever again. It was the whole shebang: Kernel panics, inconsistent views, data loss, very slow performance, split-brain problems all the time. Our set up IIRC was very simple: two bricks in a replicated volume. It worked so poorly that we had to take it out of production. Some of our experience can be explained by GlusterFS performing poorly under network partitions, but nothing could justify kernel panics. It blew my mind that Redhat acquired that company and product. Edit: I hope there's been a big improvement to the reliability and performance of GlusterFS. Can anyone with more recent experience running it in production comment?
- illumin8 11y ago
  I'm not a GlusterFS expert, and haven't used it before, but you should know that most consensus algorithms (Paxos, Raft, etc) only function reliably with an odd number of nodes. I have to wonder if your problems were mostly self-inflicted from having 2 nodes. Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). In a 3-node cluster, any system with a decent consensus algorithm (to be clear, I'm not sure if GlusterFS has one) would know that during a partition the cluster can only continue to operate if at least 2 nodes can communicate with each other to elect a new master.
  - tveita 11y ago
    > Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.
  - gamegod 11y ago
    I didn't do the initial configuration myself but I wouldn't rule out this kind of gross, self-inflicted problem. One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)
jpgvm 11y ago
GlusterFS is basically synonymous with pain. Use at your own peril.
cdnsteve 11y ago
File storage in 2016? Why not just use S3? If file storage even a problem anymore?
- Nux 11y ago
  Yes, it's still a big problem.
- ssmoot 11y ago
  Because if you have apps that weren't written at a time that S3 pricing made any sense at all (it wasn't that long ago you could afford to buy a new file-server and JBOD every month for what it would've cost to store a few dozen terabytes on S3) migration is a huge burden and expense. Really looking forward to EFS to take a big bite out of that commitment since it has reasonable pricing for projects with smaller storage requirements. Just switching a portion (photos) of the project with the most files to S3 took on the order of 800 (mostly developer) hours. At some point you have to decide what you can do with the hand you're dealt and wether you want to distribute those tens of thousands of dollars to your employees, pursuing new business or your own pocket; or reinvest it into an existing project despite having no mandate to do so. It's not an easy or cut and dry a decision to make (even today) for legacy projects IME. If you're starting a new project. Yes. S3 is the most reliable service in the AWS stack (AFAIK) by a big margin. The pricing is very reasonable today unless you're in the petabyte range maybe, and you have no operations overhead. Agreed it's a no-brainer.
- epistasis 11y ago
  Anytime you use the word "just", you're sweeping a huge number of assumptions under the rug. For example, what if you have to run code that depends on a POSIX interface? That "just" becomes a massive rewrite project.
- milesward 11y ago
  Software that talks POSIX tends to dislike object storage. Disclaimer: I work on GCP.
- boulos 11y ago
  Not mentioned yet is that nearly all object stores (S3, GCS, etc.) are pretty awful for small, random reads and writes. For workloads like rendering, the renderer is often reading a small tile of a texture (an EXR or tiff) with about 32x32 pixels (so anywhere between 1 KiB and 16 KiB depending on precision and compression). Object stores on the other hand are tuned for big files and so often work in 1 to 4 MiB chunks at best. Disclaimer: I work on Compute Engine.
Nux 11y ago
Gluster is not for the faint of heart, but as far as distributed filesystems go it's probably the easiest to set up and deal with. We've been using it in production for a few years now and having a single namespace that can basically grow ad infinitum has been pretty neat. If you want a trouble free Gluster experience stay away from MANY small files and replicated volumes.
goodcjw2 11y ago
Basically, GlusterFS is trying to solve a hard problem: make distributed/remote filesystem to feel like a local filesystem for applications built on top of it. For the client, you can choose from NFS, SMB or its homemake fuse client, which makes the remote system accessible as if everything is on local file system. I used to build similar systems in house and find it extremely painful to design and maintain, we did lots of custom hacks to make our system to suit our need. GlusterFS, as a general solution, won't have that much flexibility and may or may not suit your custom needs. Overall, I feel AWS S3 is a better (or at least simpler) approach. Just acknowledge that files are not locally stored and use them as is. AWS is experimenting EFS as well, which we found not as desirable as well. Edit: I am not saying that you cannot make GlusterFS or EFS perform great. My appoint it that it's hard to do so, and might not worth the effort to develop such a system given that S3 can serve most needs of distributed file storage.
- k_bx 11y ago
  There's Riak S2, which can give you S3-compatible API but still run on your hardware with your configs.
- spydum 11y ago
  aren't you comparing apples to oranges? S3 is an object store (and non POSIX.. Also only eventually consistent).. GlusterFS is neither of those. They simply solve different problem spaces
  - ej_campbell 11y ago
    Parent is saying that it might be simpler for many applications to forgo the need for a file system abstraction and just use S3's more limited API directly. That way, you don't get bit by thorny edge cases like "ls" of 10K files taking orders of magnitude more time than you'd expect give you're used to the speed of a local disk.
melted 11y ago
I wonder why they even did this. They already have a state of the art distributed filesystem (Colossus) which doesn't have any scalability problems at all, since they use it for everything. http://www.highlyscalablesystems.com/3202/colossus-successor-to-google-file-system-gfs/ http://www.highlyscalablesystems.com/3202/colossus-successor...
- wmf 11y ago
  Can you mount Colossus? Does it have POSIX semantics?
  - melted 11y ago
    I'm sure this could be retrofitted even if it does not.
objectivefs 11y ago
If you are looking for a POSIX compatible file system for GCE or EC2, we think our ObjectiveFS[1] is the easiest way to get started and use. It is a log structured filesystem using GCS or S3 for backend storage and with a ZFS like interface to manage your filesystems. [1] https://objectivefs.com https://objectivefs.com
stevenking86 11y ago
nice
amelius 11y ago
I'm wondering if there exist better (simpler) solutions than GlusterFS for the case where files are strictly write-once.
deleted 11y ago
[deleted]
profeta 11y ago
you can see why google is such a good marketing company. None of the links in the article is not to their own products. So, here is the link to the star of the show: https://www.redhat.com/en/technologies/storage/gluster https://www.redhat.com/en/technologies/storage/gluster

227 points by Sami_Lehtinen 11y ago

I'm not sure what is announced here. Gluster FS is for few years already (version 3.6 now), while the article doesn't mention that there is started any managed service based on it. It is more like reminder that you can set up distributed file system on your cloud servers using Gluster. Not even any step-by-step tutorial how to do that.

baldfat 11y ago
> Google Cloud Platform and Red Hat are proud to announce the availability of Red Hat Gluster Storage on Google Compute Engine. Gluster is available not announcing that this is a new technology.
- prohor 11y ago
  How this availability will be implemented then? I can install it now, so it is already available for me. Will they provide a service like Amazon Elastic File System? Or there will be some pre-configured images?
  - bonzini 11y ago
    On AWS you get pre-configured AMIs, I suppose it's similar.
- ssmoot 11y ago
  Not sure what you're saying?
- profmonocle 11y ago
  That's what I find confusing. Was it not available before now? Couldn't you just spin up some Red Hat instances and run it? It's not clear what's actually being announced - it doesn't sound like a managed service.
dragonwriter 11y ago
> I'm not sure what is announced here. I think, though I'm not sure, that its an official, active support relationship between Google and Red Hat for GlusterFS on GCE, related to Red Hat's vendor certification programs. If so, this seems like the kind of thing that has the most impact on people with support contracts with Red Hat, but should also have peripheral impacts on the stability, quality of support, etc., of the product on the GCE platform generally.
milesward 11y ago
Disclaimer: I work on GCP. Fixed that for you, check out this: https://cloud.google.com/solutions/filers-on-compute-engine https://cloud.google.com/solutions/filers-on-compute-engine

secopdev 11y ago

pricing?

milesward 11y ago
Disclaimer: I work on GCP. GlusterFS works best on RHEL, and consumes normal GCP resources like GCE and PD-SSD storage. To host a rocking fast, best practices, HA, 3TB all-SSD filer, it'd be less than $900 on GCP: https://cloud.google.com/products/calculator/#id=e76e9a5a-bf21-4ccd-9cab-5590c77394db https://cloud.google.com/products/calculator/#id=e76e9a5a-bf...

jqueryin 11y ago

Before reading the article, I was going to ask if it solves the "high read access of many small files" I/O problem, but alas, it's on GlusterFS, so only insomuch as Gluster has been making improvements these last few minor releases. Is anyone here running a GlusterFS setup with high read/write volume on small files successfully? If so, what's your secret?

cgarrigue 11y ago

It's a bit light for a press release. Considering RedHat is officially promoting AWS on their website, providing more information to let people know whether the offering on Google Cloud will be better or similar would have been better.

justinclift 11y ago
Yeah, it seems to be missing the "click here for next steps / further information" bit, which is less than optimal. :(
- milesward 11y ago
  Disclaimer: I work on GCP Yup, we goofed. Here's the next step bits: https://cloud.google.com/solutions/filers-on-compute-engine https://cloud.google.com/solutions/filers-on-compute-engine
  - justinclift 11y ago
    Ugh. So it's literally still RH Gluster, needing manual setup. Also, there's a sentence on the end of the GlusterFS section there saying: "If you want to deploy a Red Hat Gluster Storage cluster on Compute Engine, see this white paper for instructions on how to provision a multi-node cluster that includes cross-zone and cross-region replication:" Apart from the typo (pedant alert!), the URL on "white paper" goes to a non-public document only Red Hat subscribers have access to. That should probably be that fixed, so non-Red-Hat-subscribers can read the doc and know what they'll need to do up front. If people need to subscribe to RH in order to get that info... just to know what they need to do... that's probably going to hinder adoption. Potentially by a lot. ;)

pilif 11y ago

Last time I tried GlusterFS was in 2012. The way it worked was very impressive back then and I would have loved to actually put it into production. Unfortunately, I hit a roadblock in relation to enumeration of huge directories: Even with just 5K files in a directory, performance started to drop really badly to the point where enumerating a directory containing 10K files would take longer than 5 minutes. Yes. You're not supposed to store many files in a directory, but this was about giving third parties FTP upload access for product pictures and I can't possibly ask them to follow any schema for file and folder naming. These people want a directory to put stuff to with their GUI FTP client and they want their client to be able to not upload files if the target already exists. So having all files in one directory was a huge improvement UX-wise. So in the end, I had to move to nfs on top of drbd to provide shared backend storage. Enumerating 20K files over NFS still isn't fast but completes within 2 seconds instead of more than 5 minutes. Of course, now that we're talking about GlusterFS, I wonder whether this has been fixed since?

mikaelj 11y ago
Couldn't your FTP server have handled this in a clever way? For example, sort in directories by first letter, then by first two letters. While still providing a virtual flat view to the FTP client user. It'd be a simple mapping away.
- pilif 11y ago
  You assume I wanted to write an FTP server. I don't. I am using stock vsftp with a PAM module that allows authentication against our web application.
jweir 11y ago
I started using Gluster this last fall. At first it was not a contender - the articles I read were not encouraging. However many of this articles were older and it appears a lot of progress has been made with Gluster since version 3.5 It maybe worth a second look - however a directory with a large file count still might be a problem

justinclift 11y ago

Heyas, Ex-GlusterFS person here (used to work at Red Hat on the project side, leaving mid last year). "Small file access", and "lots of files in a directory" have been a pain point with GlusterFS for ages. The 3.7.0 release had some important improvements in it, specificially designed to help fix that: https://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf https://www.gluster.org/community/documentation/index.php/Fe... The latest Gluster release is 3.7.8 (the same series as 3.7.0), and is worth looking at if you're needing a good distributed file system. If you have something like 1Mill files in a single directory though... hrmmm... NFS or other technologies might still be a better idea. ;)

DannoHung 11y ago
What makes having a lot of files in a directory a hard problem?
- justinclift 11y ago
  From memory, it was something about not having a central metadata server combined with needing to talk with each of the nodes to get their idea of "the latest version" of file info. Especially tricky when taking into account edge cases, such as when some nodes are failed/down/etc. Unfortunately, I don't remember most of the details. It's fallen out of my headspace in the months since I've left. ;)
- TallGuyShort 11y ago
  It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does both extremely well is non-trivial. You can optimize to store a few very large files or many very small files, but doing both while scaling reliably is harder than you might think.
  - DannoHung 11y ago
    Naively it seems weird since metadata is just data, right? So if you can store a large amount of data in a file, you should be able to store the metadata But I don't know shit about filesystem design, soooooo...

godzillabrennus 11y ago

Glad to see Gluster is still making waves. I was an early customer. It's impressive when a brand survives acquisition much less a transition into a new type of offering like this. Kudos to everyone who helped make Gluster special!

thrownaway2424 11y ago

I see that the GlusterFS FAQ says it is fully POSIX compliant. That's a pretty good trick. Ten years ago or so I had a suite of compliance tests I would use to embarrass salesmen from iBrix and Panasas. The only actually POSIX-compliant distributed filesystem I could find in those days was Lustre (unrelated to Gluster, despite the naming). Lustre works well but it almost impossible to install and operate.

kbenson 11y ago
I remember at around the same time I was playing with GlusterFS, and it was interesting, but painful to configure as well. It was prone to have client/server configuration mismatches, and you had to be careful to reason about where you enabled a feature, the client or server, to make sure it failed in a safe way. It did have capabilities for Fcntl and flock file locking though, which was interesting at the time. Unfortunately ist was also somewhat unstable, and I would see segfaults every few weeks. My focus for work projects moved into different areas, and I didn't keep up with it's development, so I'm not sure what they've done in the last decade, but it was promising and refreshingly new back then. I should take another look.
batbomb 11y ago
In HPC/HTC, Lustre is very common, especially in DOE labs. I've never tried to install it, but I don't think my colleagues who have are some kind of special genius.
- jpgvm 11y ago
  Most HPC labs run distributions that go to quite a lot of pain to ensure Lustre works well. The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues. That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install. Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park. Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.

chatmasta 11y ago

I needed a shared volume across multiple EC2 instances in a VPC. My use case is that multiple "ingress" boxes write files to the shared volume, and then a single "worker" box processes those files. This is a somewhat unusual use case in that it means one box is responsible for 99% of IO heavy operations, and the other boxes are responsible only for writing to the volume, with no latency requirements. My solution was to mount an EBS on the "worker box," along with an NFS server. Each "ingress box" runs an NFS client that connects to the server via its internal VPC IP address, and mounts the NFS volume to a local directory. It works wonderfully. In three months of running this setup, I've had no downtime or issues, not even minor ones. Granted I don't need any kind of extreme I/O performance, so I haven't measured it, but this system took less than an hour to setup and fit my needs perfectly.

helper 11y ago
Amazon now has an NFS service (in preview release) called EFS: https://aws.amazon.com/efs/ https://aws.amazon.com/efs/

deleted 11y ago

[deleted]

Beldur 11y ago

Red Hat announcement: http://www.redhat.com/en/about/press-releases/red-hat-unveils-flexible-and-portable-cloud-storage-red-hat-gluster-storage-google-cloud-platform http://www.redhat.com/en/about/press-releases/red-hat-unveil...

gamegod 11y ago

I worked with a GlusterFS deployment in production about 2 years ago, and it was such a nightmare that I both feel compelled to write about it and never touch anything made by that team ever again. It was the whole shebang: Kernel panics, inconsistent views, data loss, very slow performance, split-brain problems all the time. Our set up IIRC was very simple: two bricks in a replicated volume. It worked so poorly that we had to take it out of production. Some of our experience can be explained by GlusterFS performing poorly under network partitions, but nothing could justify kernel panics. It blew my mind that Redhat acquired that company and product. Edit: I hope there's been a big improvement to the reliability and performance of GlusterFS. Can anyone with more recent experience running it in production comment?

illumin8 11y ago
I'm not a GlusterFS expert, and haven't used it before, but you should know that most consensus algorithms (Paxos, Raft, etc) only function reliably with an odd number of nodes. I have to wonder if your problems were mostly self-inflicted from having 2 nodes. Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). In a 3-node cluster, any system with a decent consensus algorithm (to be clear, I'm not sure if GlusterFS has one) would know that during a partition the cluster can only continue to operate if at least 2 nodes can communicate with each other to elect a new master.
- tveita 11y ago
  > Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.
- gamegod 11y ago
  I didn't do the initial configuration myself but I wouldn't rule out this kind of gross, self-inflicted problem. One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)

jpgvm 11y ago

GlusterFS is basically synonymous with pain. Use at your own peril.

cdnsteve 11y ago

File storage in 2016? Why not just use S3? If file storage even a problem anymore?

Nux 11y ago
Yes, it's still a big problem.
ssmoot 11y ago
Because if you have apps that weren't written at a time that S3 pricing made any sense at all (it wasn't that long ago you could afford to buy a new file-server and JBOD every month for what it would've cost to store a few dozen terabytes on S3) migration is a huge burden and expense. Really looking forward to EFS to take a big bite out of that commitment since it has reasonable pricing for projects with smaller storage requirements. Just switching a portion (photos) of the project with the most files to S3 took on the order of 800 (mostly developer) hours. At some point you have to decide what you can do with the hand you're dealt and wether you want to distribute those tens of thousands of dollars to your employees, pursuing new business or your own pocket; or reinvest it into an existing project despite having no mandate to do so. It's not an easy or cut and dry a decision to make (even today) for legacy projects IME. If you're starting a new project. Yes. S3 is the most reliable service in the AWS stack (AFAIK) by a big margin. The pricing is very reasonable today unless you're in the petabyte range maybe, and you have no operations overhead. Agreed it's a no-brainer.
epistasis 11y ago
Anytime you use the word "just", you're sweeping a huge number of assumptions under the rug. For example, what if you have to run code that depends on a POSIX interface? That "just" becomes a massive rewrite project.
milesward 11y ago
Software that talks POSIX tends to dislike object storage. Disclaimer: I work on GCP.
boulos 11y ago
Not mentioned yet is that nearly all object stores (S3, GCS, etc.) are pretty awful for small, random reads and writes. For workloads like rendering, the renderer is often reading a small tile of a texture (an EXR or tiff) with about 32x32 pixels (so anywhere between 1 KiB and 16 KiB depending on precision and compression). Object stores on the other hand are tuned for big files and so often work in 1 to 4 MiB chunks at best. Disclaimer: I work on Compute Engine.

Nux 11y ago

Gluster is not for the faint of heart, but as far as distributed filesystems go it's probably the easiest to set up and deal with. We've been using it in production for a few years now and having a single namespace that can basically grow ad infinitum has been pretty neat. If you want a trouble free Gluster experience stay away from MANY small files and replicated volumes.

goodcjw2 11y ago

Basically, GlusterFS is trying to solve a hard problem: make distributed/remote filesystem to feel like a local filesystem for applications built on top of it. For the client, you can choose from NFS, SMB or its homemake fuse client, which makes the remote system accessible as if everything is on local file system. I used to build similar systems in house and find it extremely painful to design and maintain, we did lots of custom hacks to make our system to suit our need. GlusterFS, as a general solution, won't have that much flexibility and may or may not suit your custom needs. Overall, I feel AWS S3 is a better (or at least simpler) approach. Just acknowledge that files are not locally stored and use them as is. AWS is experimenting EFS as well, which we found not as desirable as well. Edit: I am not saying that you cannot make GlusterFS or EFS perform great. My appoint it that it's hard to do so, and might not worth the effort to develop such a system given that S3 can serve most needs of distributed file storage.

k_bx 11y ago
There's Riak S2, which can give you S3-compatible API but still run on your hardware with your configs.
spydum 11y ago
aren't you comparing apples to oranges? S3 is an object store (and non POSIX.. Also only eventually consistent).. GlusterFS is neither of those. They simply solve different problem spaces
- ej_campbell 11y ago
  Parent is saying that it might be simpler for many applications to forgo the need for a file system abstraction and just use S3's more limited API directly. That way, you don't get bit by thorny edge cases like "ls" of 10K files taking orders of magnitude more time than you'd expect give you're used to the speed of a local disk.

melted 11y ago

I wonder why they even did this. They already have a state of the art distributed filesystem (Colossus) which doesn't have any scalability problems at all, since they use it for everything. http://www.highlyscalablesystems.com/3202/colossus-successor-to-google-file-system-gfs/ http://www.highlyscalablesystems.com/3202/colossus-successor...

wmf 11y ago
Can you mount Colossus? Does it have POSIX semantics?
- melted 11y ago
  I'm sure this could be retrofitted even if it does not.

objectivefs 11y ago

If you are looking for a POSIX compatible file system for GCE or EC2, we think our ObjectiveFS[1] is the easiest way to get started and use. It is a log structured filesystem using GCS or S3 for backend storage and with a ZFS like interface to manage your filesystems. [1] https://objectivefs.com https://objectivefs.com

stevenking86 11y ago

nice

amelius 11y ago

I'm wondering if there exist better (simpler) solutions than GlusterFS for the case where files are strictly write-once.

deleted 11y ago

[deleted]

profeta 11y ago

you can see why google is such a good marketing company. None of the links in the article is not to their own products. So, here is the link to the star of the show: https://www.redhat.com/en/technologies/storage/gluster https://www.redhat.com/en/technologies/storage/gluster