5 ms·

Not much tech content here. They racked a dense array of drives in a JBOD. There's no description of a complex or sophisticated ingestion engine, system for dis

Not much tech content here. They racked a dense array of drives in a JBOD. There's no description of a complex or sophisticated ingestion engine, system for distributing data across the array, file system, error correction, nothing. An Exabyte of disk is definitely eyebrow-raising, but companies like LinkedIn had an exabyte of storage in 2021 just in their HDFS clusters. https://www.linkedin.com/blog/engineering/open-source/the-exabyte-club-linkedin-s-journey-of-scaling-the-hadoop-distr https://www.linkedin.com/blog/engineering/open-source/the-ex...

squigz 2y ago
It's hard for me to grok that LinkedIn (or any company, really) has more than 1 exabyte of analytical data.
- Scaevolus 2y ago
  That's about 1GB per user. If you imagine them storing every interaction anyone has with the site-- every file upload, every page load, every mouse movement-- and throw in some duplication-- it's not completely impossible.
  - dmbche 2y ago
    Isn't 1gb per user absurdly large?
    - Teever 2y ago
      Makes you wonder how much facebook and google have per user.
    - woleium 2y ago
      not for gmail, or photos, but it does seem that way for linkedin, probably, unless they like storing uncompressed logs.
  - 0cf8612b2e1e 2y ago
    Do these logs ever get pruned? Is it worth knowing that 10 years ago, Johnny took 1.5 seconds to click the upvote button? Or is it just easier to keep it, imagining some what-if value extraction?
Melkman 2y ago
More information on the software side of CERN storage can be had here: https://indico.mathrice.fr/event/143/contributions/174/attachments/211/252/julien_collet-ceph_at_cern.pdf https://indico.mathrice.fr/event/143/contributions/174/attac...
rthnbgrredf 2y ago
If I remember correctly, JBOD means that if a drive fails the data on the drive is lost. If you have so many disks then you have an expected fail rate of roughly 1.7% per year (according to the popular drive statistics from back blaze). Isn't it a bit harsh to lose 1.7% of data every year? Why not a dead simple RAID that can tolerate the loss of a disk without data loss?
- ysleepy 2y ago
  Redundancy is handled at a higher layer. They are probably using ceph on top.
- russelg 2y ago
  A JBOD is as the initialism, they're Just a Bunch of Disks. They are simply enclosures that allow a large number of disks to be hooked up to a server. Any RAID or whatnot would be done on the software side, with Ceph or something similar as another reply states.

squigz 2y ago

It's hard for me to grok that LinkedIn (or any company, really) has more than 1 exabyte of analytical data.

Scaevolus 2y ago
That's about 1GB per user. If you imagine them storing every interaction anyone has with the site-- every file upload, every page load, every mouse movement-- and throw in some duplication-- it's not completely impossible.
- dmbche 2y ago
  Isn't 1gb per user absurdly large?
  - Teever 2y ago
    Makes you wonder how much facebook and google have per user.
  - woleium 2y ago
    not for gmail, or photos, but it does seem that way for linkedin, probably, unless they like storing uncompressed logs.
- 0cf8612b2e1e 2y ago
  Do these logs ever get pruned? Is it worth knowing that 10 years ago, Johnny took 1.5 seconds to click the upvote button? Or is it just easier to keep it, imagining some what-if value extraction?

Melkman 2y ago

More information on the software side of CERN storage can be had here: https://indico.mathrice.fr/event/143/contributions/174/attachments/211/252/julien_collet-ceph_at_cern.pdf https://indico.mathrice.fr/event/143/contributions/174/attac...

rthnbgrredf 2y ago

If I remember correctly, JBOD means that if a drive fails the data on the drive is lost. If you have so many disks then you have an expected fail rate of roughly 1.7% per year (according to the popular drive statistics from back blaze). Isn't it a bit harsh to lose 1.7% of data every year? Why not a dead simple RAID that can tolerate the loss of a disk without data loss?

ysleepy 2y ago
Redundancy is handled at a higher layer. They are probably using ceph on top.
russelg 2y ago
A JBOD is as the initialism, they're Just a Bunch of Disks. They are simply enclosures that allow a large number of disks to be hooked up to a server. Any RAID or whatnot would be done on the software side, with Ceph or something similar as another reply states.