Some notes from the OP author. ;) (1) Neither PVFS2 nor Lustre requires shared storage except to do hot failover for a failed server. I've personally run bot

Some notes from the OP author. ;) (1) Neither PVFS2 nor Lustre requires shared storage except to do hot failover for a failed server. I've personally run both in configurations where the only storage was node-local RAM. Disk failures can be avoided using host-based RAID, while node failures can be handled either by software disk mirroring (e.g. DRBD) or by physically moving the disks to another server. Are these approaches optimal, or even acceptable in a typical production environment? Of course not; there's a severe performance hit for the first and an even worse availability hit for the second. I do believe this is a positive differentiator for GlusterFS (though the AFR translator does exact a high performance toll) but it's not quite accurate to say or imply Lustre or PVFS2 can't be used without shared storage and it's a bit unfair to mention that for one but not the other. (2) It's only somewhat surprising to me that Lustre still doesn't have distributed metadata. I knew they were making a very big push for it in 1.8. Then I left SiCortex, and after two years dealing with Lustre lameness I actively avoided it thereafter. Adding distributed metadata after the fact is Very Hard, so the fact that they failed (for at least the third time) isn't a surprise.