4 ms·

This is neat. I like that it's using stable off-the-shelf unix components. I'm putting together the new https://neocities.org https://neocities.org fileserver

This is neat. I like that it's using stable off-the-shelf unix components. I'm putting together the new https://neocities.org https://neocities.org fileserver stuff right now, so I'll definitely be looking into this. The current plan is to use hourly rsyncs, and then implement this (or some flavor of it): http://code.google.com/p/lsyncd/ http://code.google.com/p/lsyncd/ RE inotify vs FUSE: The former is event-driven from an API, the latter I believe uses lower level blocks. Which one is better here is entirely debatable. Gluster uses a similar approach this does I believe. I'm not an expert on unix file APIs, so take all of this with a grain of salt. The biggest reason we can't use Gluster replication is that if you request a file when replicating, it goes to ask all the servers if the file is on them, instead of just failing because it's not on the local system. That's fine for many things, but it's an instant DDoS if someone runs siege on the server and just blasts you with random bunk 404 requests. You can't cache your way out of that one. Apparently the performance for instant request for lots of small files can be pretty slow too. SSHFS (and rsync using SSH) blow S3 out of the water on performance for remote filesystem work. The difference is pretty insane.

pmoriarty 12y ago
"it's an instant DDoS if someone runs siege on the server and just blasts you with random bunk 404 requests" Would you run your fileservers on a publically accessible network? If not, then the above attack would require the attackers have access to your private, internal network and at that point it could be argued that you've already lost. Of course, it's good to design defense-in-depth, so that you could survive a hit on your internal network as well. But I'm not sure how much sleep I'd lose over the possibility of a DDOS attack against a private network. On the other hand, if your servers are on a private network, I'm not sure why you'd use SFS over something like NFS.
- emeraldd 12y ago
  In this context a web-server is just a fancy interface to a file server. So you have a bunch of read-only "file servers" sitting on the public web talking to a private read/write back end.
vidarh 12y ago
Gluster does what you configure it to do, and unless you have a crazy setup it won't do what you think it will do. For your type of setup, you'd combine the "distribute" and "replicate" translators. The "distribute" translator spreads files out over sub-volumes by hashed path. Unless you specifically enable "lookup-unhashed" it will not forward requests for files that are not found to any other sub-volumes than the one matching the hash. The underlying volumes would then be "replicate" and you'd there decide how many replicas you want. There are other cluster translators than distribute too, that provides more complex distribution methods, but there you do run into overheads with determining which subvolume holds the files. You'd also usually layer the io-cache translator in (which you can add on both the clients and the servers, and which provides in memory caching), so while there are possible pathological cases, Gluster is pretty resilient. Also Gluster will replicate directly between servers to heal, but read/writes using the fuse fs will have the client talk directly to each backend and write in parallel to each member of the replicate set for that file, so there normally is no separate replication unless a server is/was down or a disk has crashed: normally, once a write is reported as complete, the file is on all replicas. But frankly almost regardless of file storage solution, your best bet is still to put Nginx or another suitable reverse proxy in front of the file servers, so you can e.g. cache the hottest requested files in faster/more expensive ways (memcached, or on SSDs)

pmoriarty 12y ago

"it's an instant DDoS if someone runs siege on the server and just blasts you with random bunk 404 requests" Would you run your fileservers on a publically accessible network? If not, then the above attack would require the attackers have access to your private, internal network and at that point it could be argued that you've already lost. Of course, it's good to design defense-in-depth, so that you could survive a hit on your internal network as well. But I'm not sure how much sleep I'd lose over the possibility of a DDOS attack against a private network. On the other hand, if your servers are on a private network, I'm not sure why you'd use SFS over something like NFS.

emeraldd 12y ago
In this context a web-server is just a fancy interface to a file server. So you have a bunch of read-only "file servers" sitting on the public web talking to a private read/write back end.

vidarh 12y ago

Gluster does what you configure it to do, and unless you have a crazy setup it won't do what you think it will do. For your type of setup, you'd combine the "distribute" and "replicate" translators. The "distribute" translator spreads files out over sub-volumes by hashed path. Unless you specifically enable "lookup-unhashed" it will not forward requests for files that are not found to any other sub-volumes than the one matching the hash. The underlying volumes would then be "replicate" and you'd there decide how many replicas you want. There are other cluster translators than distribute too, that provides more complex distribution methods, but there you do run into overheads with determining which subvolume holds the files. You'd also usually layer the io-cache translator in (which you can add on both the clients and the servers, and which provides in memory caching), so while there are possible pathological cases, Gluster is pretty resilient. Also Gluster will replicate directly between servers to heal, but read/writes using the fuse fs will have the client talk directly to each backend and write in parallel to each member of the replicate set for that file, so there normally is no separate replication unless a server is/was down or a disk has crashed: normally, once a write is reported as complete, the file is on all replicas. But frankly almost regardless of file storage solution, your best bet is still to put Nginx or another suitable reverse proxy in front of the file servers, so you can e.g. cache the hottest requested files in faster/more expensive ways (memcached, or on SSDs)