7 ms·

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

acchow 3y ago
10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.
- colechristensen 3y ago
  You can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.
  - icedchai 3y ago
    What about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.
    - fh973 3y ago
      Use a distributed file system or object store with what you need, like Quobyte or Ceph.
    - colechristensen 3y ago
      Plenty of tools for that on premises. Are you actually asking?
      - icedchai 3y ago
        No, I'm just mentioning these things to show that the decision is more complex than a "4U server" vs "S3." S3/AWS gives you a ton of stuff you'd have to implement yourself. Plus you need to hire more people to manage it.
        colechristensen 3y ago
        Minio does most of S3 for example, duh you would have to do some stuff yourself but you wouldn’t be implementing everything yourself from scratch, far from it. https://min.io/ https://min.io/
    - kazen44 3y ago
      half of these are already managed by something like ZFS.. (the only one i can think of which is not inherent in the Filesystem is High availability). But one should build that kind of redundancy on a higher level anyways. (multiple storage pools, replicating writes etc). or, if you go fibre channel, just replicate the writes across multiple storage systems and be done with it. (this requires a fibre channel network though, which is neither easy nor cheap).
- simonjgreen 3y ago
  Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of mirrors" (RAID10) gives you the safest and the highest performance array (at scales where these decisions are being made which most people would see anyway). The common argument against is the cost (usableCapacity = totalCapacity/2) but my counter argument is that RAID5 is a timebomb with a secret countdown. RAID5 is great if you don't need the data on the array. Source: >20 years running complex large arrays
  - buildbot 3y ago
    Or “software” raid 10 with ZFS mirror dev’s all slapped into one pool?
    - simonjgreen 3y ago
      Precisely that
  - wongarsu 3y ago
    What's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?
    - zokier 3y ago
      With Ceph, and I imagine other similar solutions, you can skip RAID altogether and manage redundancy at higher level over plain disks. RAID makes sense if you specifically want to have conventional filesystem on top of it, but for object storage its not necessary.
      - simonjgreen 3y ago
        You're applying the same principles, just at the object level rather than the filesystem level. You still have to choose a redundancy pattern. But I do agree with you, it does make the problem significantly easier/different. ZFS and Ceph are marvelous creations.
- aljarry 3y ago
  But that's on a single RAID in a single location. You'd probably need at least one failover, and backups for that as well, if you'd like to target higher availability / durability. S3 keeps multiple copies in multiple geographic locations.
  - jlokier 3y ago
    When you are distributing over multiple geographic locations, and much of the data can tolerate a longer round trip time for access, you can extend the RAID pattern across multiple locations, instead of within a location. Provided you have enough network bandwidth for recovery, you don't need a full RAID at each individual site. Having more locations is actually helpful for this. Because of the independence of different locations, the risk of data loss is in some ways lower than a local RAID system where all drives might be fried at the same time by a single event.
withinboredom 3y ago
Buy hard drives. Typically the spinning ones for the data because $/GB is really cheap. Then you buy some SSDs for your actual applications to run on. From there you install Hadoop, maybe a query tool like Spark, Impala, and Presto. Then you install some reporting tools like Apache SuperSet. At least that's how we did it at my old job with Petabytes of data.
zokier 3y ago
Quickly checking Dell sells you boxes that are advertised as 14PB per rack and have S3 api. You probably can do better by shopping around, but it gives a ballpark what is available in the on-prem market.
pjc50 3y ago
Have a look at the Backblaze blogs, where they detail how they built a storage provider on the cheap.
- bluedino 3y ago
  and be prepared to write your own software stack
dx034 3y ago
If you don't mind it running in a data center in Europe, you could rent servers from Hetzner for ca. $400 per month for 220TB. I'm sure they can provide a few PB on the spot. Spread them across their locations and you have geo redundancy too. Next to no egress fee also allows to have a copy with another provider. https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

colechristensen 3y ago
You can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.
- icedchai 3y ago
  What about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.
  - fh973 3y ago
    Use a distributed file system or object store with what you need, like Quobyte or Ceph.
  - colechristensen 3y ago
    Plenty of tools for that on premises. Are you actually asking?
    - icedchai 3y ago
      No, I'm just mentioning these things to show that the decision is more complex than a "4U server" vs "S3." S3/AWS gives you a ton of stuff you'd have to implement yourself. Plus you need to hire more people to manage it.
      - colechristensen 3y ago
        Minio does most of S3 for example, duh you would have to do some stuff yourself but you wouldn’t be implementing everything yourself from scratch, far from it. https://min.io/ https://min.io/
  - kazen44 3y ago
    half of these are already managed by something like ZFS.. (the only one i can think of which is not inherent in the Filesystem is High availability). But one should build that kind of redundancy on a higher level anyways. (multiple storage pools, replicating writes etc). or, if you go fibre channel, just replicate the writes across multiple storage systems and be done with it. (this requires a fibre channel network though, which is neither easy nor cheap).
simonjgreen 3y ago
Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of mirrors" (RAID10) gives you the safest and the highest performance array (at scales where these decisions are being made which most people would see anyway). The common argument against is the cost (usableCapacity = totalCapacity/2) but my counter argument is that RAID5 is a timebomb with a secret countdown. RAID5 is great if you don't need the data on the array. Source: >20 years running complex large arrays
- buildbot 3y ago
  Or “software” raid 10 with ZFS mirror dev’s all slapped into one pool?
  - simonjgreen 3y ago
    Precisely that
- wongarsu 3y ago
  What's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?
  - zokier 3y ago
    With Ceph, and I imagine other similar solutions, you can skip RAID altogether and manage redundancy at higher level over plain disks. RAID makes sense if you specifically want to have conventional filesystem on top of it, but for object storage its not necessary.
    - simonjgreen 3y ago
      You're applying the same principles, just at the object level rather than the filesystem level. You still have to choose a redundancy pattern. But I do agree with you, it does make the problem significantly easier/different. ZFS and Ceph are marvelous creations.
aljarry 3y ago
But that's on a single RAID in a single location. You'd probably need at least one failover, and backups for that as well, if you'd like to target higher availability / durability. S3 keeps multiple copies in multiple geographic locations.
- jlokier 3y ago
  When you are distributing over multiple geographic locations, and much of the data can tolerate a longer round trip time for access, you can extend the RAID pattern across multiple locations, instead of within a location. Provided you have enough network bandwidth for recovery, you don't need a full RAID at each individual site. Having more locations is actually helpful for this. Because of the independence of different locations, the risk of data loss is in some ways lower than a local RAID system where all drives might be fried at the same time by a single event.

withinboredom 3y ago

Buy hard drives. Typically the spinning ones for the data because $/GB is really cheap. Then you buy some SSDs for your actual applications to run on. From there you install Hadoop, maybe a query tool like Spark, Impala, and Presto. Then you install some reporting tools like Apache SuperSet. At least that's how we did it at my old job with Petabytes of data.

zokier 3y ago

Quickly checking Dell sells you boxes that are advertised as 14PB per rack and have S3 api. You probably can do better by shopping around, but it gives a ballpark what is available in the on-prem market.

pjc50 3y ago

Have a look at the Backblaze blogs, where they detail how they built a storage provider on the cheap.

bluedino 3y ago
and be prepared to write your own software stack

dx034 3y ago

If you don't mind it running in a data center in Europe, you could rent servers from Hetzner for ca. $400 per month for 220TB. I'm sure they can provide a few PB on the spot. Spread them across their locations and you have geo redundancy too. Next to no egress fee also allows to have a copy with another provider. https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx