7 ms·

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don

by kondro 2y ago

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

bayindirh 2y ago
If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
- zaphirplane 2y ago
  > or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
  - bayindirh 2y ago
    These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal. Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand. These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.
- Closi 2y ago
  Resilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.
Neil44 2y ago
At the moment they're just paying someone else to buy $5000 SSD's and run a database on them at many X markup.
- omeid2 2y ago
  There is no upper bounds to economy of scales. Maybe there is for the cents per GB of raw storage, but power usage, security, rent, and everything else scales too, and few of them have upper bounds on economy of scales.
  - Retric 2y ago
    Economies of scale generally have upper limits. Often when you approach the largest scale the existing market will supply you essentially need to become your own supplier which then runs into span of control issues. The organization needs to become competitive in that new market or their costs increase. Keep scaling and eventually vertical integration ends up looking like a Soviet style planned economy. Your remote mining town needs some way for people to get soap etc so you open a store with it’s own supply chain etc etc.
    - omeid2 2y ago
      Yes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.
      - Retric 2y ago
        Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented. But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the phone launches. They have such tight integration not just with the manufacturer’s process but also their finances because Apple simply can’t get replacements at scale on short notice. And remember that’s at ~30% market share, it just gets worse after that.
ndriscoll 2y ago
There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.
- mobilemidget 2y ago
  Do you have any good recommendation for such 1U server with 32 slots? Thanks
  - perryh2 2y ago
    Supermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3D1U%26drive%3D32 https://www.supermicro.com/en/products/nvme?pro=formfactor%3...
    - hallway_monitor 2y ago
      Shows 2U in order to fit 32 drives, but hopefully you can find a 2U slot for this absolute monster of a storage box if you have that need.
      - ndriscoll 2y ago
        This one is 1U and supports dual socket CPU and 6 TB RAM: https://www.supermicro.com/en/products/system/1U/1029/SSG-1029P-NEL32R.cfm https://www.supermicro.com/en/products/system/1U/1029/SSG-10... I've never done procurement so can't really speak to recommendations. I just see that the products apparently exist. There's also 32 drive 1U JBOF enclosures for more expansion: https://www.supermicro.com/en/products/system/1u/136/ssg-136r-nel32jbf.cfm https://www.supermicro.com/en/products/system/1u/136/ssg-136...
- jakjak123 2y ago
  Our ops team actually wanted to do this, but we on the project have nightmares from putting 1PB of database on a single host ><
choppaface 2y ago
StorageReview plays with 2PB flash machines all the time https://www.youtube.com/watch?v=UQMKtlIjeuk https://www.youtube.com/watch?v=UQMKtlIjeuk 1PB in a rack with spinning rust + flash buffer has been easy for years now.

by kondro 2y ago

bayindirh 2y ago

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.

zaphirplane 2y ago
> or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
- bayindirh 2y ago
  These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal. Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand. These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.
Closi 2y ago
Resilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.

Neil44 2y ago

At the moment they're just paying someone else to buy $5000 SSD's and run a database on them at many X markup.

omeid2 2y ago
There is no upper bounds to economy of scales. Maybe there is for the cents per GB of raw storage, but power usage, security, rent, and everything else scales too, and few of them have upper bounds on economy of scales.
- Retric 2y ago
  Economies of scale generally have upper limits. Often when you approach the largest scale the existing market will supply you essentially need to become your own supplier which then runs into span of control issues. The organization needs to become competitive in that new market or their costs increase. Keep scaling and eventually vertical integration ends up looking like a Soviet style planned economy. Your remote mining town needs some way for people to get soap etc so you open a store with it’s own supply chain etc etc.
  - omeid2 2y ago
    Yes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.
    - Retric 2y ago
      Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented. But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the phone launches. They have such tight integration not just with the manufacturer’s process but also their finances because Apple simply can’t get replacements at scale on short notice. And remember that’s at ~30% market share, it just gets worse after that.

ndriscoll 2y ago

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

mobilemidget 2y ago
Do you have any good recommendation for such 1U server with 32 slots? Thanks
- perryh2 2y ago
  Supermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3D1U%26drive%3D32 https://www.supermicro.com/en/products/nvme?pro=formfactor%3...
  - hallway_monitor 2y ago
    Shows 2U in order to fit 32 drives, but hopefully you can find a 2U slot for this absolute monster of a storage box if you have that need.
    - ndriscoll 2y ago
      This one is 1U and supports dual socket CPU and 6 TB RAM: https://www.supermicro.com/en/products/system/1U/1029/SSG-1029P-NEL32R.cfm https://www.supermicro.com/en/products/system/1U/1029/SSG-10... I've never done procurement so can't really speak to recommendations. I just see that the products apparently exist. There's also 32 drive 1U JBOF enclosures for more expansion: https://www.supermicro.com/en/products/system/1u/136/ssg-136r-nel32jbf.cfm https://www.supermicro.com/en/products/system/1u/136/ssg-136...
jakjak123 2y ago
Our ops team actually wanted to do this, but we on the project have nightmares from putting 1PB of database on a single host ><

choppaface 2y ago

StorageReview plays with 2PB flash machines all the time https://www.youtube.com/watch?v=UQMKtlIjeuk https://www.youtube.com/watch?v=UQMKtlIjeuk 1PB in a rack with spinning rust + flash buffer has been easy for years now.