Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be ca

Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be cached in ram, because it's small", yes, but then you'll end up querying many nodes just for metadata queries. Yes it's nicer to manage only 1 system, but in big scenarios it's probably better to separate. You can have 50+TB of nvme in 1 server, so your metadata layer probably doesn't need to horizontally scale. Imagine if you lose some objects (because you lost some replicas). You won't even know WHICH objects you lost, because the metadata is gone together with the data. Having separate, you can add a 5-replicas to metadata to be even safer compared to the usual 3 replicas.