7 ms·

What's the best way to store around 300TiB data cost-effectively? Assuming would have access from time to time to around 25% of that data (monthly) through 1Gbp

by liuliu 3y ago

What's the best way to store around 300TiB data cost-effectively? Assuming would have access from time to time to around 25% of that data (monthly) through 1Gbps connection. On-prem seems to be the best, but carrying around 30 disks is no small feat. Edit: The data will not be mutated after write. Might append new data occasionally after the first dump.

jeffbee 3y ago
The main questions would be durability and availability. Maybe only because I am scarred by experience but "on-prem" seems to me quite obviously the worst possible way to do it.
- ulnarkressty 3y ago
  Why not both? Store it locally for ease of access, maybe with a solid state cache, backup in the cloud for durability. Storage is cheap nowadays. The more interesting question is availability - at the current HDD sizes, "classic" RAID is not sufficient.
  - Dylan16807 3y ago
    > at the current HDD sizes, "classic" RAID is not sufficient. Eh, RAID 6 is fine. Your chance of losing two extra drives in a two day rebuild isn't that high if you've been testing your drives regularly. And if you haven't been testing regularly, even small drives would eventually have some scary risks of dying. That said, 50% more drives with raidz3 is a lot safer than RAID 6.
    - geoah 3y ago
      It’s not as clear cut as this. Raid6 on how many disks? raid6/zraid2 with 6 disks total might be fine, but 20 disks is not. One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt.
      - Dylan16807 3y ago
        > Raid6 on how many disks? The same number of disks as you were using when the disks were smaller. I'm comparing like-for-like except disk size. > One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt. Which is why I mentioned regular testing. If your disks are on the verge of death it doesn't matter how much parity you have or how big they are, a rebuild has too much chance of failure. You need to make sure they can handle a moderately heavy load. But if you're putting them under that load regularly, a rebuild is very unlikely to be the last straw for multiple drives.
gauravphoenix 3y ago
What's the access pattern? how often you write and read? and how much?
willcodeforfoo 3y ago
As far as I know, even with the recent increase, Backblaze B2 is still the most cost efficient option at $6/TB/mo... but I'll be watching replies closely.
- secabeen 3y ago
  iDrive E2 is $4/TB/mo, that's the cheapest I've seen from a known vendor: https://www.idrive.com/object-storage-e2/ https://www.idrive.com/object-storage-e2/
  - deleted 3y ago
    [deleted]
- nottorp 3y ago
  Is there some 3rd party client for their backup service for linux? I was just looking at them for offsite but I have mac and linux boxes to back up, not mac and windows.
suprfsat 3y ago
You could probably put the disks in an enclosure and then you'd only have 1, maybe 2 items to carry around.
deleted 3y ago
[deleted]
croisillon 3y ago
have you tried getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem?
- tornato7 3y ago
  An FTP account where?
  - antisthenes 3y ago
    You can create unlimited FTP accounts locally in a config file.
wintogreen74 3y ago
how often is the data being appended or modified? You might consider a blended solution with the most cost effective on prem-disks w/ no durability or backup, then writing any additions to cold storage which is cheap. The only time you need to access that is (hopefully rare) restore scenarios.
jolmg 3y ago
> carrying around 30 disks is no small feat. 20-22TB HDDs are available nowadays, so more like 14-15 disks.
tornato7 3y ago
The datahoarder subreddit is probably the best place to ask for this info.
- geoah 3y ago
  It really isn’t. They are very hostile against people who has been using/abusing google/amazon/dropbox for storing insane amounts of data. https://www.reddit.com/r/DataHoarder/comments/155so84/dropbox_business_advanced_not_unlimited_anymore/ https://www.reddit.com/r/DataHoarder/comments/155so84/dropbo... https://www.reddit.com/r/DataHoarder/comments/15gf2rc/dropbox_now_limiting_advanced_plans_to_1tb_per/ https://www.reddit.com/r/DataHoarder/comments/15gf2rc/dropbo...
deadbunny 3y ago
rsync.net

by liuliu 3y ago

jeffbee 3y ago

The main questions would be durability and availability. Maybe only because I am scarred by experience but "on-prem" seems to me quite obviously the worst possible way to do it.

ulnarkressty 3y ago
Why not both? Store it locally for ease of access, maybe with a solid state cache, backup in the cloud for durability. Storage is cheap nowadays. The more interesting question is availability - at the current HDD sizes, "classic" RAID is not sufficient.
- Dylan16807 3y ago
  > at the current HDD sizes, "classic" RAID is not sufficient. Eh, RAID 6 is fine. Your chance of losing two extra drives in a two day rebuild isn't that high if you've been testing your drives regularly. And if you haven't been testing regularly, even small drives would eventually have some scary risks of dying. That said, 50% more drives with raidz3 is a lot safer than RAID 6.
  - geoah 3y ago
    It’s not as clear cut as this. Raid6 on how many disks? raid6/zraid2 with 6 disks total might be fine, but 20 disks is not. One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt.
    - Dylan16807 3y ago
      > Raid6 on how many disks? The same number of disks as you were using when the disks were smaller. I'm comparing like-for-like except disk size. > One of the more common scenarios with consumer grade hardware is a degraded array getting completely lost while being rebuilt. Which is why I mentioned regular testing. If your disks are on the verge of death it doesn't matter how much parity you have or how big they are, a rebuild has too much chance of failure. You need to make sure they can handle a moderately heavy load. But if you're putting them under that load regularly, a rebuild is very unlikely to be the last straw for multiple drives.

gauravphoenix 3y ago

What's the access pattern? how often you write and read? and how much?

willcodeforfoo 3y ago

As far as I know, even with the recent increase, Backblaze B2 is still the most cost efficient option at $6/TB/mo... but I'll be watching replies closely.

secabeen 3y ago
iDrive E2 is $4/TB/mo, that's the cheapest I've seen from a known vendor: https://www.idrive.com/object-storage-e2/ https://www.idrive.com/object-storage-e2/
- deleted 3y ago
  [deleted]
nottorp 3y ago
Is there some 3rd party client for their backup service for linux? I was just looking at them for offsite but I have mac and linux boxes to back up, not mac and windows.

suprfsat 3y ago

You could probably put the disks in an enclosure and then you'd only have 1, maybe 2 items to carry around.

deleted 3y ago

[deleted]

croisillon 3y ago

have you tried getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem?

tornato7 3y ago
An FTP account where?
- antisthenes 3y ago
  You can create unlimited FTP accounts locally in a config file.

wintogreen74 3y ago

how often is the data being appended or modified? You might consider a blended solution with the most cost effective on prem-disks w/ no durability or backup, then writing any additions to cold storage which is cheap. The only time you need to access that is (hopefully rare) restore scenarios.

jolmg 3y ago

> carrying around 30 disks is no small feat. 20-22TB HDDs are available nowadays, so more like 14-15 disks.

tornato7 3y ago

The datahoarder subreddit is probably the best place to ask for this info.

geoah 3y ago
It really isn’t. They are very hostile against people who has been using/abusing google/amazon/dropbox for storing insane amounts of data. https://www.reddit.com/r/DataHoarder/comments/155so84/dropbox_business_advanced_not_unlimited_anymore/ https://www.reddit.com/r/DataHoarder/comments/155so84/dropbo... https://www.reddit.com/r/DataHoarder/comments/15gf2rc/dropbox_now_limiting_advanced_plans_to_1tb_per/ https://www.reddit.com/r/DataHoarder/comments/15gf2rc/dropbo...

deadbunny 3y ago

rsync.net