3 ms·

How is this different from Lustre (lustre.org) ?

anotherjesse 17y ago
We are using Lustre on a projec. The differences that jump out to me on http://ceph.newdream.net/about/ http://ceph.newdream.net/about/ 1) auto-balancing when new storage nodes are added 2) copies of objects stored in multiple storage nodes Running HA Lustre requires a LOT of work. 1) Lustre requires tricks like copying files to rebalance 2) Lustre only has one copy of any section, so each OST is a SPOF. Sun recommends deploying OST in pairs with DRDB and Heartbeat functionality (cutting your space in half and complicating the deploying), but a box failing won't break your FS. This doesn't fix the problem that if a network partition occurs since you still have locality of chunks in a single rack. For more information about how complicated Lustre in a production environment is check out the talks at the last UG by Sun employees (Lessons Learned & Best Practices, Managing High Availability on a Shine-Equipped Lustre Cluster): http://wiki.lustre.org/index.php/Lustre_User_Group_2009 I've not tested Ceph in production but many people I talk to about our Lustre woes recommend Ethan Miller's work - http://users.soe.ucsc.edu/~elm/ http://users.soe.ucsc.edu/~elm/ - which leads to Ceph ;) Even if you aren't worrying about availability, Lustre has issues with usage: Lustre requires older kernels - which means compatibility with modern non-RHEL distros leads to headaches when you need the lastest version of KVM (or python). Lustre also tries to be a POSIX filesystem but it breaks horrible when you try to use features like O_APPEND from multiple nodes (file corruption!) We are still tracking down Lustre breaking when we read files in a certain order (rsync is ok, but start skipping around in a file read leads to the kernel stopping responding) After months of dealing with Lustre issues with trying to support POSIX I have come to love the non-POSIX (S3-like API) to distributed filesystems.

We are using Lustre on a projec. The differences that jump out to me on http://ceph.newdream.net/about/ http://ceph.newdream.net/about/ 1) auto-balancing when new storage nodes are added 2) copies of objects stored in multiple storage nodes Running HA Lustre requires a LOT of work. 1) Lustre requires tricks like copying files to rebalance 2) Lustre only has one copy of any section, so each OST is a SPOF. Sun recommends deploying OST in pairs with DRDB and Heartbeat functionality (cutting your space in half and complicating the deploying), but a box failing won't break your FS. This doesn't fix the problem that if a network partition occurs since you still have locality of chunks in a single rack. For more information about how complicated Lustre in a production environment is check out the talks at the last UG by Sun employees (Lessons Learned & Best Practices, Managing High Availability on a Shine-Equipped Lustre Cluster): http://wiki.lustre.org/index.php/Lustre_User_Group_2009 I've not tested Ceph in production but many people I talk to about our Lustre woes recommend Ethan Miller's work - http://users.soe.ucsc.edu/~elm/ http://users.soe.ucsc.edu/~elm/ - which leads to Ceph ;) Even if you aren't worrying about availability, Lustre has issues with usage: Lustre requires older kernels - which means compatibility with modern non-RHEL distros leads to headaches when you need the lastest version of KVM (or python). Lustre also tries to be a POSIX filesystem but it breaks horrible when you try to use features like O_APPEND from multiple nodes (file corruption!) We are still tracking down Lustre breaking when we read files in a certain order (rsync is ok, but start skipping around in a file read leads to the kernel stopping responding) After months of dealing with Lustre issues with trying to support POSIX I have come to love the non-POSIX (S3-like API) to distributed filesystems.