3 ms·

Exactly. The only time the cloud services really pay off are when you need to scale up or down massive capacity overnight. The odds of that happening at most co

Exactly. The only time the cloud services really pay off are when you need to scale up or down massive capacity overnight. The odds of that happening at most companies are very, very slim, even most of the web companies. If you're really concerned about that though, I'd say go for the cloud option - just make it so you can start small on a single physical server, but can scale onto the cloud, then migrate that to physical hardware as needed. With this set up, you save massive cash and avoid vendor lock-in.

mrep 9y ago
Or if you are a team like mine that develops/manages 200 servers, 15 RDS database instances, 500 TB of compressed S3 data, and provides accessibility to all of that through API's with only 5 developers because of how easy AWS makes it for us. While our server costs are probably higher than going bare metal, how many developers would it take you to manage all of that on bare metal hosts?
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- cookiecaper 9y ago
  The expense of one's outlay to Amazon is not a persuasive statistic. It only shows that you have a lot of incentive to keep confirmation bias in gear. AWS is specifically designed to maximize instance count (that is, cost). The reason "devops" has exploded since EC2 hit critical mass is that before EC2, people were reasonable with the number of servers they needed. With EC2, it's all water, and it's so easy to press "create instance" that people just do it without thinking, and wonder how they ended up paying Amazon $100k/mo for something that used to cost them $5k/mo. I'm engaged on a project now that has similar figures to what you've quoted here. It could be done with less than two dozen bare-metal servers. I know because it was done with less than two dozen bare-metal servers before someone in the C suite felt left out and suggested some "modernization" via the cloud. I think that cloud zealots are mostly people who were either deathly afraid of sysadmin or people who were cutting their teeth as cloud got hyped, because there's no way any competent person who was doing this type of stuff before 2008-2009 can pretend that Amazon is not laughing all the way to the bank. This is not to say that cloud has no advantages or that its use is always inappropriate, but what you're describing is pure fantasy. Yes, a small team of coders should be more than capable of managing the servers that they need, especially if they are renting dedicated boxes from a professional datacenter facility that handles hardware swaps and similar failures for them.
  - mrep 9y ago
    Our EC2 costs are about the same as S3, RDS is the most as I forgot to mention the backup servers we have in each region but we are trying to move all of that data to S3, and more than our SQS costs. I'm pretty new to the backend design cost/benefit (have only used AWS since I graduated college) but I'm curious to here about bare metal storage solutions for 500 TB of compressed json as I have not read any compelling solutions for that data requirement (Outside of MongoDB but I have heard good and bad things about that).
    - cookiecaper 9y ago
      In 2017, 500TB is easily within the realm of colocated bare metal. There are many build it yourself options, which would involve things similar to this SuperMicro JBOD [0] + some other servers/external RAID controller to handle distributions over the disks. You could also go really barebones and do a couple of home-built "RAIDZilla"-style devices. [1] For the cost of one month's S3 storage, you can buy a pre-built Backblaze Pod from a third-party supplier [2]. I've found this to be the case with Amazon; the monthly cost is about 50%-100% of the permanent cost for the actual hardware (which will usually last at least 3-5 years). Even if you have to hire a couple of your own hardware jockeys, you're going to be saving 6x. For a less DIY route, any SAN provider will be able to accommodate 1PB (for redundancy) without breaking a sweat. Of course, this will be a large upfront expenditure, but it should still easily be cheaper than S3 over the long run. There are tons of options and storage engineering is a big field. Look around and I'm sure you'll find something acceptable. ----- All this said, I really have to be skeptical that you need 500TB of (compressed!) JSON data. I would look into how much of that data you really need to keep, set up some retention policies, and seriously consider reworking your storage format to make these numbers more reasonable (JSON is obviously not space-efficient), which will not only greatly reduce infrastructure costs but also make the project much easier to handle. You may also wish to look into modern compression codecs if you haven't already. LZMA provides the best ratio but the compression cycle is slow (decompression is fast). Brotli and zstd are new compression options that are at least comparable to gzip in ratio and much faster. Data deduplication should also help. [0] https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K23JBOD-rack-mountable-4U/4415857.aspx https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K2... [1] https://www.glaver.org/raidzilla25/ https://www.glaver.org/raidzilla25/ [2] https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... (estimates cost of third-party pod at $12,849.40; S3 cost calculator says 500TB of storage in us-east-1 is $12,407.17 without any bandwidth, request costs)
- deleted 9y ago
  [deleted]

mrep 9y ago

Or if you are a team like mine that develops/manages 200 servers, 15 RDS database instances, 500 TB of compressed S3 data, and provides accessibility to all of that through API's with only 5 developers because of how easy AWS makes it for us. While our server costs are probably higher than going bare metal, how many developers would it take you to manage all of that on bare metal hosts?

deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
cookiecaper 9y ago
The expense of one's outlay to Amazon is not a persuasive statistic. It only shows that you have a lot of incentive to keep confirmation bias in gear. AWS is specifically designed to maximize instance count (that is, cost). The reason "devops" has exploded since EC2 hit critical mass is that before EC2, people were reasonable with the number of servers they needed. With EC2, it's all water, and it's so easy to press "create instance" that people just do it without thinking, and wonder how they ended up paying Amazon $100k/mo for something that used to cost them $5k/mo. I'm engaged on a project now that has similar figures to what you've quoted here. It could be done with less than two dozen bare-metal servers. I know because it was done with less than two dozen bare-metal servers before someone in the C suite felt left out and suggested some "modernization" via the cloud. I think that cloud zealots are mostly people who were either deathly afraid of sysadmin or people who were cutting their teeth as cloud got hyped, because there's no way any competent person who was doing this type of stuff before 2008-2009 can pretend that Amazon is not laughing all the way to the bank. This is not to say that cloud has no advantages or that its use is always inappropriate, but what you're describing is pure fantasy. Yes, a small team of coders should be more than capable of managing the servers that they need, especially if they are renting dedicated boxes from a professional datacenter facility that handles hardware swaps and similar failures for them.
- mrep 9y ago
  Our EC2 costs are about the same as S3, RDS is the most as I forgot to mention the backup servers we have in each region but we are trying to move all of that data to S3, and more than our SQS costs. I'm pretty new to the backend design cost/benefit (have only used AWS since I graduated college) but I'm curious to here about bare metal storage solutions for 500 TB of compressed json as I have not read any compelling solutions for that data requirement (Outside of MongoDB but I have heard good and bad things about that).
  - cookiecaper 9y ago
    In 2017, 500TB is easily within the realm of colocated bare metal. There are many build it yourself options, which would involve things similar to this SuperMicro JBOD [0] + some other servers/external RAID controller to handle distributions over the disks. You could also go really barebones and do a couple of home-built "RAIDZilla"-style devices. [1] For the cost of one month's S3 storage, you can buy a pre-built Backblaze Pod from a third-party supplier [2]. I've found this to be the case with Amazon; the monthly cost is about 50%-100% of the permanent cost for the actual hardware (which will usually last at least 3-5 years). Even if you have to hire a couple of your own hardware jockeys, you're going to be saving 6x. For a less DIY route, any SAN provider will be able to accommodate 1PB (for redundancy) without breaking a sweat. Of course, this will be a large upfront expenditure, but it should still easily be cheaper than S3 over the long run. There are tons of options and storage engineering is a big field. Look around and I'm sure you'll find something acceptable. ----- All this said, I really have to be skeptical that you need 500TB of (compressed!) JSON data. I would look into how much of that data you really need to keep, set up some retention policies, and seriously consider reworking your storage format to make these numbers more reasonable (JSON is obviously not space-efficient), which will not only greatly reduce infrastructure costs but also make the project much easier to handle. You may also wish to look into modern compression codecs if you haven't already. LZMA provides the best ratio but the compression cycle is slow (decompression is fast). Brotli and zstd are new compression options that are at least comparable to gzip in ratio and much faster. Data deduplication should also help. [0] https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K23JBOD-rack-mountable-4U/4415857.aspx https://www.cdw.com/shop/products/Supermicro-SC417-BE1C-R1K2... [1] https://www.glaver.org/raidzilla25/ https://www.glaver.org/raidzilla25/ [2] https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv... (estimates cost of third-party pod at $12,849.40; S3 cost calculator says 500TB of storage in us-east-1 is $12,407.17 without any bandwidth, request costs)
deleted 9y ago
[deleted]