4 ms·

Our EC2 costs are about the same as S3, RDS is the most as I forgot to mention the backup servers we have in each region but we are trying to move all of that d

by mrep 10y ago

Our EC2 costs are about the same as S3, RDS is the most as I forgot to mention the backup servers we have in each region but we are trying to move all of that data to S3, and more than our SQS costs. I'm pretty new to the backend design cost/benefit (have only used AWS since I graduated college) but I'm curious to here about bare metal storage solutions for 500 TB of compressed json as I have not read any compelling solutions for that data requirement (Outside of MongoDB but I have heard good and bad things about that).

cookiecaper 10y ago
In 2017, 500TB is easily within the realm of colocated bare metal. There are many build it yourself options, which would involve things similar to this SuperMicro JBOD [0] + some other servers/external RAID controller to handle distributions over the disks. You could also go really barebones and do a couple of home-built "RAIDZilla"-style devices. [1] For the cost of one month's S3 storage, you can buy a pre-built Backblaze Pod from a third-party supplier [2]. I've found this to be the case with Amazon; the monthly cost is about 50%-100% of the permanent cost for the actual hardware (which will usually last at least 3-5 years). Even if you have to hire a couple of your own hardware jockeys, you're going to be saving 6x. For a less DIY route, any SAN provider will be able to accommodate 1PB (for redundancy) without breaking a sweat. Of course, this will be a large upfront expenditure, but it should still easily be cheaper than S3 over the long run. There are tons of options and storage engineering is a big field. Look around and I'm sure you'll find something acceptable. ----- All this said, I really have to be skeptical that you need 500TB of (compressed!) JSON data. I would look into how much of that data you really need to keep, set up some retention policies, and seriously consider reworking your storage format to make these numbers more reasonable (JSON is obviously not space-efficient), which will not only greatly reduce infrastructure costs but also make the project much easier to handle. You may also wish to look into modern compression codecs if you haven't already. LZMA provides the best ratio but the compression cycle is slow (decompression is fast). Brotli and zstd are new compression options that are at least comparable to gzip in ratio and much faster. Data deduplication should also help. [0] https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K23JBOD-rack-mountable-4U/4415857.aspx https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K2... [1] https://www.glaver.org/raidzilla25/ https://www.glaver.org/raidzilla25/ [2] https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... (estimates cost of third-party pod at $12,849.40; S3 cost calculator says 500TB of storage in us-east-1 is $12,407.17 without any bandwidth, request costs)
chx 10y ago
Grab small servers with one or two big disk (Hetzner will give you two 4TB disks for 39.00 EUR, OVH one 6TB for about 30 USD w/ an ARM CPU, less than 50 USD for two 4TB under the soyoustart brand), serve them over iSCSI, use RAID-Z3 to create a nice huge ZFS filesystem. Do not forget to have hot spares. Done. Costs about half or even third of what S3 costs and it's a filesystem instead of an object storage. Also has transparent compression. ZFS grows online. Both OVH and Hetzner and I bet other such providers have API so you can just script the whole thing to grow as soon as you cross a threshold.

by mrep 10y ago

cookiecaper 10y ago

In 2017, 500TB is easily within the realm of colocated bare metal. There are many build it yourself options, which would involve things similar to this SuperMicro JBOD [0] + some other servers/external RAID controller to handle distributions over the disks. You could also go really barebones and do a couple of home-built "RAIDZilla"-style devices. [1] For the cost of one month's S3 storage, you can buy a pre-built Backblaze Pod from a third-party supplier [2]. I've found this to be the case with Amazon; the monthly cost is about 50%-100% of the permanent cost for the actual hardware (which will usually last at least 3-5 years). Even if you have to hire a couple of your own hardware jockeys, you're going to be saving 6x. For a less DIY route, any SAN provider will be able to accommodate 1PB (for redundancy) without breaking a sweat. Of course, this will be a large upfront expenditure, but it should still easily be cheaper than S3 over the long run. There are tons of options and storage engineering is a big field. Look around and I'm sure you'll find something acceptable. ----- All this said, I really have to be skeptical that you need 500TB of (compressed!) JSON data. I would look into how much of that data you really need to keep, set up some retention policies, and seriously consider reworking your storage format to make these numbers more reasonable (JSON is obviously not space-efficient), which will not only greatly reduce infrastructure costs but also make the project much easier to handle. You may also wish to look into modern compression codecs if you haven't already. LZMA provides the best ratio but the compression cycle is slow (decompression is fast). Brotli and zstd are new compression options that are at least comparable to gzip in ratio and much faster. Data deduplication should also help. [0] https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K23JBOD-rack-mountable-4U/4415857.aspx https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K2... [1] https://www.glaver.org/raidzilla25/ https://www.glaver.org/raidzilla25/ [2] https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... (estimates cost of third-party pod at $12,849.40; S3 cost calculator says 500TB of storage in us-east-1 is $12,407.17 without any bandwidth, request costs)

chx 10y ago

Grab small servers with one or two big disk (Hetzner will give you two 4TB disks for 39.00 EUR, OVH one 6TB for about 30 USD w/ an ARM CPU, less than 50 USD for two 4TB under the soyoustart brand), serve them over iSCSI, use RAID-Z3 to create a nice huge ZFS filesystem. Do not forget to have hot spares. Done. Costs about half or even third of what S3 costs and it's a filesystem instead of an object storage. Also has transparent compression. ZFS grows online. Both OVH and Hetzner and I bet other such providers have API so you can just script the whole thing to grow as soon as you cross a threshold.