4 ms·
We are using Lustre on a projec. The differences that jump out to me on http://ceph.newdream.net/about/ http://ceph.newdream.net/about/ 1) auto-balancing when
by anotherjesse 17y ago
We are using Lustre on a projec. The differences that jump out to me on http://ceph.newdream.net/about/ http://ceph.newdream.net/about/
1) auto-balancing when new storage nodes are added
2) copies of objects stored in multiple storage nodes
Running HA Lustre requires a LOT of work.
1) Lustre requires tricks like copying files to rebalance
2) Lustre only has one copy of any section, so each OST is a SPOF. Sun recommends deploying OST in pairs with DRDB and Heartbeat functionality (cutting your space in half and complicating the deploying), but a box failing won't break your FS. This doesn't fix the problem that if a network partition occurs since you still have locality of chunks in a single rack.
For more information about how complicated Lustre in a production environment is check out the talks at the last UG by Sun employees (Lessons Learned & Best Practices, Managing High Availability on a Shine-Equipped Lustre Cluster):
http://wiki.lustre.org/index.php/Lustre_User_Group_2009
I've not tested Ceph in production but many people I talk to about our Lustre woes recommend Ethan Miller's work - http://users.soe.ucsc.edu/~elm/ http://users.soe.ucsc.edu/~elm/ - which leads to Ceph ;)
Even if you aren't worrying about availability, Lustre has issues with usage:
Lustre requires older kernels - which means compatibility with modern non-RHEL distros leads to headaches when you need the lastest version of KVM (or python).
Lustre also tries to be a POSIX filesystem but it breaks horrible when you try to use features like O_APPEND from multiple nodes (file corruption!) We are still tracking down Lustre breaking when we read files in a certain order (rsync is ok, but start skipping around in a file read leads to the kernel stopping responding)
After months of dealing with Lustre issues with trying to support POSIX I have come to love the non-POSIX (S3-like API) to distributed filesystems.