4 ms·

For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of hav

For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).

donw 15y ago
If your business depends on handling large amounts of data, you want that person on-staff anyway.
aaronjg 15y ago
Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper. Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local storage, since all of their data is generated on-site. However, even at this scale, they don't build the boxes themselves [2]. [1] http://www.genome.gov/27538886 http://www.genome.gov/27538886 [2] http://www.isilon.com/press-release/isilon-iq-powers-data-storage-next-generation-dna-sequencing http://www.isilon.com/press-release/isilon-iq-powers-data-st...
- gcb 15y ago
  He says it's 1/10 of the price, so assume 10 Colo. enough with people defending s3. Go buy a IBM mainframe already :)
- ww520 15y ago
  Exactly. Plus power and rack space are not cheap. Also hiring the sys admins 24/7 won't be cheap.
- spydum 15y ago
  Also quite important to mention, use of S3 will chew up your upstream bandwidth while making backups. Also, it will affect your recovery time objectives if you ever needed to run a restore (you will NOT be getting 200MByte/sec to or from S3, unless you have some sweet network connectivity).
buff-a 15y ago
Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If I needed this much space in a start-up I'd totally do it myself - it would be a good use of my time. No, my concern with this setup is that it looks (to me) to be a "w00t I got $6000 so lets build a 733t boxen!" It looks cool and fragile, instead of boring and robust. UPDATE: To be fair, I'm not hammering those drives continuously. How much does that change the equation? Even if it goes from 45 minutes every two years to 45 minutes per day, you'd still be net positive. Basically, you can't prevent drive failures. But if you can keep the failures restricted to the ones that just require popping in a new drive, then your "operator time" is minimal. Its when you lose an array that you're in trouble. Again: Do not use one big box!
- baddox 15y ago
  Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think of it.
  - amadvance 15y ago
    You can evaluate SnapRAID. It's a software, not realtime, RAID5/6 like solution. http://snapraid.sourceforge.net/ http://snapraid.sourceforge.net/
  - thaumaturgy 15y ago
    I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is: - A box from abmx.com. Really nice people, good hardware. - An Areca RAID controller (might have to talk to them about this). - FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community. - ZFS. This particular box has had almost no downtime in the past year, and the downtime it's had has been caused either by administration tasks (software updates or configuration modification) or a USB flash drive failure (we've been experimenting with BSD and Linux server configurations on flash media separate from the server's data storage, so that we can do things like show up with replacement or upgraded configurations, plug them in, and be done. USB flash media is not reliable enough for this though, even if you're not doing swap on it). I just added a couple of 3TB Hitachi drives to the array last week. Resizing the volume was a little fiddly because the Areca firmware needed an update first before it could recognize the 3TB drives (not a problem you'd have on newer models), but otherwise, everything happened live, while the box was up and running and doing its job. When the Areca controller software finished resizing the volume, zfs happily said, "oh, I'm bigger now! I can handle that!" OpenBSD's softraid is also pretty good stuff, but I don't think it makes sense to use it unless you're trying to cheap out with a small 2-bay box and no controller in a RAID 1 or something. SBS 2008 is not my favorite thing in the world.
WettowelReactor 15y ago
Well first off all S3 is not the only option. Hell even going mid tier with an established SAN installed at a cheap colo is way cheaper than S3 and leaps ahead of their solution.

donw 15y ago

If your business depends on handling large amounts of data, you want that person on-staff anyway.

aaronjg 15y ago

Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper. Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local storage, since all of their data is generated on-site. However, even at this scale, they don't build the boxes themselves [2]. [1] http://www.genome.gov/27538886 http://www.genome.gov/27538886 [2] http://www.isilon.com/press-release/isilon-iq-powers-data-storage-next-generation-dna-sequencing http://www.isilon.com/press-release/isilon-iq-powers-data-st...

gcb 15y ago
He says it's 1/10 of the price, so assume 10 Colo. enough with people defending s3. Go buy a IBM mainframe already :)
ww520 15y ago
Exactly. Plus power and rack space are not cheap. Also hiring the sys admins 24/7 won't be cheap.
spydum 15y ago
Also quite important to mention, use of S3 will chew up your upstream bandwidth while making backups. Also, it will affect your recovery time objectives if you ever needed to run a restore (you will NOT be getting 200MByte/sec to or from S3, unless you have some sweet network connectivity).

buff-a 15y ago

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If I needed this much space in a start-up I'd totally do it myself - it would be a good use of my time. No, my concern with this setup is that it looks (to me) to be a "w00t I got $6000 so lets build a 733t boxen!" It looks cool and fragile, instead of boring and robust. UPDATE: To be fair, I'm not hammering those drives continuously. How much does that change the equation? Even if it goes from 45 minutes every two years to 45 minutes per day, you'd still be net positive. Basically, you can't prevent drive failures. But if you can keep the failures restricted to the ones that just require popping in a new drive, then your "operator time" is minimal. Its when you lose an array that you're in trouble. Again: Do not use one big box!

baddox 15y ago
Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think of it.
- amadvance 15y ago
  You can evaluate SnapRAID. It's a software, not realtime, RAID5/6 like solution. http://snapraid.sourceforge.net/ http://snapraid.sourceforge.net/
- thaumaturgy 15y ago
  I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is: - A box from abmx.com. Really nice people, good hardware. - An Areca RAID controller (might have to talk to them about this). - FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community. - ZFS. This particular box has had almost no downtime in the past year, and the downtime it's had has been caused either by administration tasks (software updates or configuration modification) or a USB flash drive failure (we've been experimenting with BSD and Linux server configurations on flash media separate from the server's data storage, so that we can do things like show up with replacement or upgraded configurations, plug them in, and be done. USB flash media is not reliable enough for this though, even if you're not doing swap on it). I just added a couple of 3TB Hitachi drives to the array last week. Resizing the volume was a little fiddly because the Areca firmware needed an update first before it could recognize the 3TB drives (not a problem you'd have on newer models), but otherwise, everything happened live, while the box was up and running and doing its job. When the Areca controller software finished resizing the volume, zfs happily said, "oh, I'm bigger now! I can handle that!" OpenBSD's softraid is also pretty good stuff, but I don't think it makes sense to use it unless you're trying to cheap out with a small 2-bay box and no controller in a RAID 1 or something. SBS 2008 is not my favorite thing in the world.

WettowelReactor 15y ago

Well first off all S3 is not the only option. Hell even going mid tier with an established SAN installed at a cheap colo is way cheaper than S3 and leaps ahead of their solution.