5 ms·

> exabyte-scale storage system Somewhat of a random question, can you point me to some state of the art research?

by vikiomega9 7y ago

ghostpepper 7y ago
I am not the op nor an expert on this, but I think https://ceph.io/ https://ceph.io/ targets clusters of this scale.
- Mave83 7y ago
  Yes, with croit.io you can manage Ceph clusters with ease, reducing Laber costs and increase reliability. We build clusters of low PB scale that have a TCO with everything from labor to hardware, from financing to electric, from routers to cables and can be run below 3€/TB. For that you can store data in Block(rbd, iscsi), Objekt(S3, swift) or Filestorage(CephFS, NFS, SMB), high available on Supermicro hardware in any datacenter worldwide. Feel free to contact us or use our free community edition to start your own cluster.
sydney6 7y ago
Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)
- londons_explore 7y ago
  Google's clusters are all block and file based (depending on what layer you want to use them at)
- statictype 7y ago
  What does object-based mean? How is an object different from a file (which I presumed was a collection of blocks)?
  - tgsovlerkhgsel 7y ago
    My guess: No random write access to objects, you can (at best) append-only but often you can only append until the object is finalized, and cannot read it until it is finalized.
  - tw04 7y ago
    It means you access your object through a GUID. Think about it like parking your own car vs. a valet. When you park your own car you need to know the address of the garage you parked in, the floor you were on, and the spot you were in. When you valet park, you hand the attendant a ticket and he brings your car back. With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.
    - statictype 7y ago
      Got it. Thanks for explaining.
    - toolslive 7y ago
      Some object stores do this, but aws S3 for example does not. You can list the contents of buckets, nicely sorted by name. You can mimic directory structures if you want. However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.
  - toolslive 7y ago
    it's like a key-value store, (or a dictionary). However, the values are objects (big blobs of data). This means you can't update parts of objects without rewriting the blob. However, most of the object stores offer metadata operations (move, tag, ...), concat of n objects into 1 and partial reads.
wmf 7y ago
https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock... http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf http://static.googleusercontent.com/media/research.google.co... https://queue.acm.org/detail.cfm?id=1594206 https://queue.acm.org/detail.cfm?id=1594206 http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

> exabyte-scale storage system Somewhat of a random question, can you point me to some state of the art research?

ghostpepper 7y ago

I am not the op nor an expert on this, but I think https://ceph.io/ https://ceph.io/ targets clusters of this scale.

Mave83 7y ago
Yes, with croit.io you can manage Ceph clusters with ease, reducing Laber costs and increase reliability. We build clusters of low PB scale that have a TCO with everything from labor to hardware, from financing to electric, from routers to cables and can be run below 3€/TB. For that you can store data in Block(rbd, iscsi), Objekt(S3, swift) or Filestorage(CephFS, NFS, SMB), high available on Supermicro hardware in any datacenter worldwide. Feel free to contact us or use our free community edition to start your own cluster.

sydney6 7y ago

Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)

londons_explore 7y ago
Google's clusters are all block and file based (depending on what layer you want to use them at)
statictype 7y ago
What does object-based mean? How is an object different from a file (which I presumed was a collection of blocks)?
- tgsovlerkhgsel 7y ago
  My guess: No random write access to objects, you can (at best) append-only but often you can only append until the object is finalized, and cannot read it until it is finalized.
- tw04 7y ago
  It means you access your object through a GUID. Think about it like parking your own car vs. a valet. When you park your own car you need to know the address of the garage you parked in, the floor you were on, and the spot you were in. When you valet park, you hand the attendant a ticket and he brings your car back. With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.
  - statictype 7y ago
    Got it. Thanks for explaining.
  - toolslive 7y ago
    Some object stores do this, but aws S3 for example does not. You can list the contents of buckets, nicely sorted by name. You can mimic directory structures if you want. However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.
- toolslive 7y ago
  it's like a key-value store, (or a dictionary). However, the values are objects (big blobs of data). This means you can't update parts of objects without rewriting the blob. However, most of the object stores offer metadata operations (move, tag, ...), concat of n objects into 1 and partial reads.

wmf 7y ago

https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock... http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf http://static.googleusercontent.com/media/research.google.co... https://queue.acm.org/detail.cfm?id=1594206 https://queue.acm.org/detail.cfm?id=1594206 http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...