5 ms·

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in c

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens. * We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only. * If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs. * RDS is still just a dream for us, since we still don't have a comfort level with performance. * Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances. * We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS. * We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project. * If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.

pearkes 14y ago
> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project. Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?
- jeffbarr 14y ago
  Yes, please. Let me know and I'll make sure that the team sees it.
- jaytaylor 14y ago
  SNS is frequently a huge PITA to get working, often requiring SSH tunnels and the like.
  - Zombieball 14y ago
    Not sure I follow. Could you elaborate how SNS would require an SSH tunnel to work? I've never encountered this issue in environments I have worked in.
    - jaytaylor 14y ago
      I looked into this and I was actually mistaken - I was referencing SES, not SNS. Apologies!
- malachismith 14y ago
  The idea that SNS isn't first-in, first-out is scary enough to me - but the fact that it's also "at least one copy will be delivered" (rather than "only one copy") makes it kind of insane.
seldo 14y ago
My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.
jbarham 14y ago
> We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. FYI http://www.1000tb.com/ http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...
- brianhc 14y ago
  try 100TB.com
vosper 14y ago
What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win. We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers. Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!
- ceejayoz 14y ago
  I think SNS was a typo for SES.
deleted 14y ago
[deleted]
jaytaylor 14y ago
You might try using the SendHub API (https://www.sendhub.com https://www.sendhub.com) for your SMS alerts. Way easier to get started with than Twilio's, and the groups feature makes it very easy for people to subscribe to groups of alerts. Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)
ghshephard 14y ago
I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If you actually use that 100 Terabytes per server, they start to curtail/rate limit/traffic shape you. C) Some other option I haven't considered? Maybe they just rely on the vast majority of their customers not using that bandwidth, and subsidizing (significantly) those that do? blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.
- jsatok 14y ago
  I haven't done a ton of work with 100tb.com, just a couple servers with them a few years back. It's great bandwidth. They're a SoftLayer reseller, so the nextwork blend is pretty good. I never got close to the 100tb cap, so am not sure if they'd impose some metering if you're doing it on a recurring basis. I feel like most of the people I've talked to about 100tb.com have gone in with the mentality that it's nearly unlimited, but end up using less than 1tb. If you aggregate across a ton of resold servers, I bet it's possible to make the model work pretty well.
- e1ven 14y ago
  Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights. http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale-for-resellers/ http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale... I don't want to try to guess why they did that, but it's something that should never be forgotten.
  - ghshephard 14y ago
    So - that pretty much answers my question. If you actually start to use that 100TB, they drop you as a customer. TANSSAAFL, so, if it looks to good to be true ($0.33/mbit/second on Softlayer network) - then it probably is. I'd be very interested though, in what people have been able to sustain per server at 100TB without getting dropped. My guess is that it's somewhere around 32 Terabytes/month per server before 100TB gets a little angsty (100 mbits/second sustained), particularly if you have more than a dozen or so servers with them (filling out a GigE sustained)
falcolas 14y ago
> RDS is still just a dream for us In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over. RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.
deckchair 14y ago
I agree, a good post. AWS has stated that they are working on the link between EBS and ELB: https://aws.amazon.com/message/680342/ https://aws.amazon.com/message/680342/ The point in the post that the AWS management console runs on EBS is fishy...

pearkes 14y ago

> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project. Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?

jeffbarr 14y ago
Yes, please. Let me know and I'll make sure that the team sees it.
jaytaylor 14y ago
SNS is frequently a huge PITA to get working, often requiring SSH tunnels and the like.
- Zombieball 14y ago
  Not sure I follow. Could you elaborate how SNS would require an SSH tunnel to work? I've never encountered this issue in environments I have worked in.
  - jaytaylor 14y ago
    I looked into this and I was actually mistaken - I was referencing SES, not SNS. Apologies!
malachismith 14y ago
The idea that SNS isn't first-in, first-out is scary enough to me - but the fact that it's also "at least one copy will be delivered" (rather than "only one copy") makes it kind of insane.

seldo 14y ago

My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.

jbarham 14y ago

> We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. FYI http://www.1000tb.com/ http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...

brianhc 14y ago
try 100TB.com

vosper 14y ago

What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win. We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers. Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!

ceejayoz 14y ago
I think SNS was a typo for SES.

deleted 14y ago

[deleted]

jaytaylor 14y ago

You might try using the SendHub API (https://www.sendhub.com https://www.sendhub.com) for your SMS alerts. Way easier to get started with than Twilio's, and the groups feature makes it very easy for people to subscribe to groups of alerts. Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)

ghshephard 14y ago

I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If you actually use that 100 Terabytes per server, they start to curtail/rate limit/traffic shape you. C) Some other option I haven't considered? Maybe they just rely on the vast majority of their customers not using that bandwidth, and subsidizing (significantly) those that do? blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.

jsatok 14y ago
I haven't done a ton of work with 100tb.com, just a couple servers with them a few years back. It's great bandwidth. They're a SoftLayer reseller, so the nextwork blend is pretty good. I never got close to the 100tb cap, so am not sure if they'd impose some metering if you're doing it on a recurring basis. I feel like most of the people I've talked to about 100tb.com have gone in with the mentality that it's nearly unlimited, but end up using less than 1tb. If you aggregate across a ton of resold servers, I bet it's possible to make the model work pretty well.
e1ven 14y ago
Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights. http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale-for-resellers/ http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale... I don't want to try to guess why they did that, but it's something that should never be forgotten.
- ghshephard 14y ago
  So - that pretty much answers my question. If you actually start to use that 100TB, they drop you as a customer. TANSSAAFL, so, if it looks to good to be true ($0.33/mbit/second on Softlayer network) - then it probably is. I'd be very interested though, in what people have been able to sustain per server at 100TB without getting dropped. My guess is that it's somewhere around 32 Terabytes/month per server before 100TB gets a little angsty (100 mbits/second sustained), particularly if you have more than a dozen or so servers with them (filling out a GigE sustained)

falcolas 14y ago

> RDS is still just a dream for us In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over. RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.

deckchair 14y ago

I agree, a good post. AWS has stated that they are working on the link between EBS and ELB: https://aws.amazon.com/message/680342/ https://aws.amazon.com/message/680342/ The point in the post that the AWS management console runs on EBS is fishy...