16 ms·

AWS mistakes to avoid

537 points by hellomichibye 11y ago

alttab 11y ago
Also, dont use Dynamo unless you have a really good reason. "It scales better than MySQL" is not a good reason. Have fun migrating data and re-indexing constantly!
- chucky_z 11y ago
  What about as a kind of Redis stand-in? I'm curious about the DynamoDB PHP session driver. It sounds like you have a lot of experience with Dynamo. All the use-cases I seem to keep coming up with are more for storing global environment keys outside of the environment itself and using a large number of IAM roles to access individual keys, and as a kind of throwaway 'I need to store this somewhere, but it doesn't really fit in the main DB, and I still need it to persist for at least awhile' case.
  - kelnos 11y ago
    Not the grandparent, but I have a lot of experience with DynamoDB as well. I migrated a self-managed, sharded redis store to DDB close to a year ago, and what it really depends on are your read and write patterns, and whether or not you can live with the opacity of what DDB is doing behind the scenes. An example: if you provision X capacity units, you're actually provisioning X/N capacity units per partition. AWS is mostly transparent (via documentation) about when a table splits into a new partition (I say "mostly" because I was told by AWS that the numbers in the docs aren't quite right), but you'll have no idea how the keys are hashed between partitions, and you won't know if you have a hot partition that's getting slammed because your keys aren't well-distributed. Well, no, I take that back -- you will know, because you'll get throttled even though you're not consuming anywhere near the capacity you've provisioned. You just won't know how to fix it without a lot of trial and error (which isn't great in a production system). If your r/w usage isn't consistent, you'll either have periods of throttling, or you'll have to over-provision and waste money. There's no auto-scaling like you can do with EC2. Not trying to knock the product: still using DDB... but getting it to a point where I felt reasonably confident about it took way longer than managing my own data store... and then they had a 6-hour outage one early, early Sunday morning a couple months ago. Possibly solution: dual-writes to two AWS regions at once and the ability to auto-pivot reads to the backup region. Which of course doubles provisioning costs. Ok, maybe I'm knocking it a little. It's a good product, but there are definitely tradeoffs.
    - chucky_z 11y ago
      Was the ElastiCache service around when you migrated Redis to Dynamo? This is another alternative I'm considering.
      - kelnos 11y ago
        The problem with ElastiCache -- and why I rejected it as an option -- as that they make you define a 30-minute "maintenance window" where AWS can apply patches and reboot your cache instances. In practice I've heard that this happens rarely, and when it does, the downtime is short, but it can in theory cause longer outages. And in the case of both the redis and memcached backends, if the maintenance requires restarting redis/memcached or rebooting the instance, you lose all data in the cache (at least up until your last backup). For this particular project, that amount of downtime would easily cause a real outage for customers, and was unacceptable.
    - boomzilla 11y ago
      You should put some effort on designing the key schema, especially the hash key. Don't do timestamp, or sequence. I had to do a 'encrypted' key schema on my sequence IDs so sequential records are spread into different partitions. Actually it worked out well as I can exposed these encrypted keys externally too. Agreed that DDB documentation could be better on the best practices. I found this deep dive youtube video very helpful: https://www.youtube.com/watch?v=VuKu23oZp9Q https://www.youtube.com/watch?v=VuKu23oZp9Q
      - kelnos 11y ago
        Oh, absolutely. The problem is that your key schema also ties you to what kind of queries you can do efficiently. My primary keys actually are very well-distributed across the hash space, but in my initial implementation, one of the global secondary indexes had a key that caused throttling and absolutely destroyed performance. Dropping that index meant losing the ability to do an entire class of queries. In this case, we were (sorta) able to live with that, but I can imagine many cases where that's a problem. That's actually another instance of the lack of transparency and trial-and-error: "oh hey, writes are getting throttled... no idea why... let's drop this index and see if it helps".
- markism 11y ago
  Beginner here, how do you keep the database available to multiple instances without Dynamo?
  - jdimov9 11y ago
    Use RDS.
- dexterdog 11y ago
  Can you expand on your last statement?
- greenleafjacob 11y ago
  Is local secondary indices and lower operational costs than Cassandra a good reason(s)?
  - Rapzid 11y ago
    Not without considering the throttling you will hit if you exceed provisioned IOPS. You really want to consider all the angles on dynamoDB before committing to it; limitations, unique benefits, interface, etc.
  - wsh91 11y ago
    Operational costs are in the eye of the beholder. We evaluated DynamoDB for a new use case but we're going with Cassandra because the storage costs alone more than made up for ops overhead. At least IMO, Cassandra's support for multicolumn range queries (via clustering columns) and the cheaper storage you can use obviate local secondary indices. (Cassandra has secondary indices as well but it seems like most folks prefer further denormalization--at least, that's what we're doing.)
- joshuahornby 11y ago
  Can you explain these comments? I am currently developing a system using Dyanmo as the database, I need a JSON like structure as the documents need to be flexible as different documents will contain different keys. So I am interested to hear why you think Dyanmo is a bad choice.
- ap22213 11y ago
  I use Dynamo because I'm lazy. Need some JSON fast? 3 lines and no worries. DynamoDB db = new DynamoDB(new AmazonDynamoDBClient()); Table myTable = db.getTable("table"); Item myItem = myTable.getItem("id", id); Sure, I don't use it for a lot of things, but for cross-VPC / cross-account configuration, it's pretty nice.
andrioni 11y ago
I second the recommendation to use CloudFormation + packer + ELBs + auto scaling groups for web applications whenever possible, it just makes everything so easy and automatic. Of course, there's a learning curve and you pay a premium for all that automation, but in my experience it has been usually worth it so far.
- ZacharyPitts 11y ago
  I like going one step further and making deployments and AWS resources into reusable code using Troposphere: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
deleted 11y ago
[deleted]
- hellomichibye 11y ago
  +1 I concentrated on non security related mistakes. security will follow next week... :)
- startitrrr3 11y ago
  Great points siddharth_mal
kevindeasis 11y ago
You know what I've realized that's really important. More AWS tutorials is really needed. There's numerous of new programmers who want to learn AWS, but can't finish building anything because they get buried in documentation. I find there are a lot of high-level abstracted tutorials, but for the new services, there aren't a lot of detailed tutorials. For instance, an implemented cognito->gateway->lambda->dynamodb is really hard for a newbie to do.
- ninjay 11y ago
  I found this series very helpful: https://medium.com/aws-activate-startup-blog https://medium.com/aws-activate-startup-blog
- jcyw 11y ago
  I have some good experience with qwiklab. Their self-paced lab starts with UI, then turned to aws cli. They also give a nice lesson on CF.
- euphemize 11y ago
  Great point. Also, the pipeline you mention removes many risks related to the 'common' mistakes mentioned in the article (scaling, monitoring, provisioning it all done for you - or almost). We're using gateway -> lambda -> dynamodb, and there are a tons of gotchas and small things that AWS need to iron out, especially with gateway -> lambda.
  - LoSboccacc 11y ago
    Agreed. Was looking into that for a serverless model inspired by (1) but there is still loads of things they can't do/won't do 1 https://github.com/serverless/serverless https://github.com/serverless/serverless
    - ac360 11y ago
      Working on it :) We are hacking away full time on the Serverless Framework and have some big features coming in the next few days.
nodesocket 11y ago
My huge recommendation is to put production instances in a completely seperate region than development and staging instances. I actually just discovered that you can limit IAM API keys to a specific region, you just need to create a custom policy. The following policy is an example: { "Version": "2012-10-17", "Statement": [ { "Sid": "SOME-ID-HERE", "Effect": "Allow", "Action": [ "ec2:*" ], "Condition": { "StringEquals": { "ec2:Region": "us-west-2" } }, "Resource": [ "*" ] } ] }
- cheetos 11y ago
  Why?
  - nodesocket 11y ago
    If you segment production instances from develop/staging you can use IAM rules to grant specific privileges based on the entire region, instead of by tag (which is fragile). Additionally, it is less error prone when you are making manual changes in the console as it requires switching regions between production and develop/staging.
- eropple 11y ago
  Why would you do this instead of using multiple AWS accounts? Different regions have different feature sets (available instance types, beta eligibility, etc.). I strongly recommend instead using multiple AWS accounts instead and keeping them in the same region. That said, since you should be using an infrastructure provisioning tool like CloudFormation, the tagging solution should not be a particularly big obstacle.
  - kbar13 11y ago
    yes, use multiple accounts. you can use STS to grant permissions between the accounts if needed.
ninjay 11y ago
What are the current ways to make creating CloudFormation templates not so painful?
- loki77 11y ago
  You should check out troposphere (https://github.com/cloudtools/troposphere/ https://github.com/cloudtools/troposphere/) and stacker (https://github.com/remind101/stacker/ https://github.com/remind101/stacker/). I'm a maintainer of both - we try to make CF easier by catching errors earlier, and allowing you to do things like write for loops, in troposphere. stacker tries to take your troposphere templates and tie them together as totally separate stacks.
siddharth_mal 11y ago
I'd like to add two more: 1) Not giving out your access and secret keys in scripts/buckets. 2) Always using IAM roles with your EC2
- Bestcoderplanet 11y ago
  +1 I concentrated on non security related mistakes. security will follow next week... :)
- misiti3780 11y ago
  how do you avoid 1 - it seems impossible ?
  - mhluongo 11y ago
    IAM roles let you assign temporary credentials to machines running scripts. The machine can then hit an internal AWS URL to get the temporary credentials. Many tools know to look for these credentials by default- eg boto checks for credentials in environment variables, config files, and the machines IAM role.
    - idunno246 11y ago
      And there's a few tools to emulate the metadata service locally if you need it on dev laptops which makes it use a role as if a server
  - rodrickbrown 11y ago
    Take a look at hashicorp vault - https://hashicorp.com/blog/vault.html https://hashicorp.com/blog/vault.html
- hrez 11y ago
  IAM roles is ok as long as you realize that anything and anybody on that instance gets access to those credentials.
peterwaller 11y ago
I like CloudFormation. Unfortunately it is very unwieldy to write CloudFormation templates directly, and we're not about to start using the AWS CFN GUI editor! It seems like the assembly of the AWS ecosystem. Does anyone else have a favourite hammer for this particular nail? I'd love to have something better than our home-baked solution, but I'm yet to find anything which doesn't introduce other flaws, such as an incomplete implementation (missing parameters or resource types) or ultimately making a leaky abstraction on top of CloudFormation somehow. I ended up brewing a reasonably straightforward solution using Python as a (minimal) DSL which emits JSON. Its primary purpose is to support the whole of the CFN ecosystem (not just implement some small part of EC2, for instance) while also not trying to be too clever. It has about 50-100 lines of python which implements helper functions such as ref(), join() and load_user_data(), and not many other things. There is an almost 1-to-1 correspondence between the generated CFN configuration and the python source. As a bonus it checks for a few common mistakes like broken refs or parameters which aren't used. I have heard that similar solutions have been reinvented in a few places, including the BBC. But I'm yet to see a good public solution!
- otterley 11y ago
  Some people like SparkleFormation (http://www.sparkleformation.io/ http://www.sparkleformation.io/). I'll warn you, though, that it's not a good example of how to program in Ruby. It abuses method_missing to the point that it makes your implementations difficult to debug.
  - nikolay 11y ago
    I also don't like their made-up terms such as "dynamics", etc. The documentation is pretty confusing as well.
- andystanton 11y ago
  Troposphere (https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere) is a mature Python CloudFormation solution that sounds similar to your home brewed one.
kennu 11y ago
I warmly recommend the Serverless framework for building basic web applications on AWS. It handles CloudFormation details for you, but lets you customize them if needed. Not suitable for every possible app though.
rgawdzik 11y ago
Check out http://convox.com/ http://convox.com/ (YC S15) for an alternative to avoiding manual infrastructure.
- nzoschke 11y ago
  Convox co-founder here. Thanks for the shout out! That's exactly right, avoid manual infrastructure. Someday we will all have something like Rails for infrastructure. Strong conventions around best practices. If you follow these conventions you can avoid bespoke or manual configurations and focus solely on your app logic. We're building Convox to advance this goal.
misiti3780 11y ago
Another, somewhat obvious one: Be extremely careful when using public customized AMIs, a lot of times ~/.ssh/authorized_hosts contains public keys and this is obviously a huge security problem
- kbar13 11y ago
  no, public keys are not a security problem.
  - joseph 11y ago
    They are if you don't know who has the corresponding private key.
    - kbar13 11y ago
      oh right when using public AMIs. I thought parent meant publishing AMIs. derp.
  - iancarroll 11y ago
    But the fact they allow access to your system is.
  - mhurron 11y ago
    Public keys on your host where you don't control who has the private key are.
  - deleted 11y ago
    [deleted]
- nodesocket 11y ago
  Yeah, I almost feel like AWS should clear the contents of authoried_keys for each user when you make an AMI public. That of course is bound to break some things, but prevents security oversights.
  - brianwawok 11y ago
T3RMINATED 11y ago
By choosing AWS to begin with you have commited the Top 1 Mistake that will come to haunt you.
buremba 11y ago
Use OpsWorks if possible. It's free and provides a simple interface that allows you to deploy/upgrade your apps automatically and monitors your instances automatically using CloudWatch.
- wsh91 11y ago
  +1. AWS has done an amazing job with OpsWorks. It's free Chef 12. What's not to love? (The only drawback I've noticed so far is some standard Chef stuff--vault in particular--not working. Other than that, very pleased.)
diziet 11y ago
Something major is missing: Running Demand instances instead of Reserved
- hrez 11y ago
  Or running Reserved instead of onDemand. It's all about use cases.
tedmiston 11y ago
I'll add one that was especially common for people coming off the year of free tier a couple years ago. I'm not sure that AWS has changed it yet. 6. Not starting a box/instance/database and forgetting it's running until you receive the bill after your free tier expires.
- molecule 11y ago
  I expected to see something like this generalized in the article as 6. Not setting up billing alarms, http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...
ommunist 11y ago
I attended aws workshop, several years ago, there I realized, that this vast ecosystem of infrastructure services require a popularisation effort of similar scale. And not just in plain English. IAM that days was not very clearly understandable, search supported only ascii and was not really documented, now that ecosystem is in order of magnitude larger and more complex. And efforts like cloudonaut's should be greatly appreciated. For the greater public good. Thank you, man!
dantiberian 11y ago
The biggest mistake I've seen with AWS (and committed myself), is not reading the manuals for the services you're using. While some people complain about the AWS manuals not being complete, there is still a lot of good information in there that you might miss if you're just clicking through the console.
elwell 11y ago
I'd like to add, if using Elastic Beanstalk, don't directly attach an RDS instance when creating the environment. If you do, you won't be able to destroy your environment without also deleting the RDS instance. Instead, create the RDS instance separately, and just add the proper security group for the environment to be able to access the host. Then you can easily create a new eb environment with any config changes (there are some config changes you can't make to an eb environment without creating a new one from scratch) and then connect to your existing db.
- gwintrob 11y ago
  Is there a site that collects these sorts of tips? Seems like there are a lot of potential pitfalls that can be hard to find in docs like: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-rds.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-r...
- softdev12 11y ago
  interesting. i was under the impression you could just create a snapshot (before destroying the environment) and then restore as needed. am i missing something? Edit: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R...
  - sdroy 11y ago
    If you can take the database instance offline and if you do not need to access it from outside of the beanstalk app then that is fine.
vacri 11y ago
> There is no reason why you should manage your infrastructure manually. It's unprofessional! It's a mess! Nonsense. Cloudformation has it's issues. It takes time to learn and implement. The templates can break, requiring the stack to be destroyed and remade. In the sample in the article, the database is in the same template as everything else - what fun that will be when an update breaks the template and you have to reapply the stack (which destroys the existing database). Cloudformation is good, but it comes with caveats, and the idea that you should only manage an AWS stack with CF is utter tripe. As with everything, it depends on your use case. Also weird is the article's demand of using autoscaling groups to monitor single instances. Why not just monitor them directly with cloudwatch? > There is no reason - beside manually managed infrastructure - to not decrease the instance size (number of machines or c3.xlarge to c3.large) if you realize that your EC2 instances are underutilized. This is wrong, too. Autoscaling takes time to scale up, and it scales up in stages. If you get sudden traffic, autoscaling can take too long. Again, it's about knowing your use case. Unfortunately for us, we can get sudden traffic when one of our clients does a media release and they don't tell us ahead of time, The five or so minutes it takes for instances to trigger the warning, start up a new set, and then attach these to a load balancer is too long for this particular use case, so we just have to run with a certain amount of excess capacity. Autoscaling is awesome, but this article is way too didactic in it's No True Scotsman approach.
- Rapzid 11y ago
  Cloudformation can really be a harsh mistress. I feel I'm constantly discovering good reasons for not including certain sets of resources in the same templates for reasons you and others touch on. And nested stacks? You know, never say never.. Probably never again.
voltagex_ 11y ago
I find IAM particularly difficult to use - I feel like there should be a button to create a user/group that can do only X, Y, Z. I realise policy templates get most of the way there but I still had to go and read the syntax for them because DescribeRegions wasn't in the list I needed. I'm also not sure how to make the jump from exporting AWS_ACCESS_KEY_ID and having my instances automatically request the permissions they need - STS?
- inopinatus 11y ago
  If your code is using the AWS SDK then you don't need to export an access key for your running code. Just create an EC2 Service Role in IAM with the policy you want attached to the role; then launch the instance with that role, to get automatic temporary credentials. Refs: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use... http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles...
- cheeseprocedure 11y ago
  > I'm also not sure how to make the jump from exporting AWS_ACCESS_KEY_ID and having my instances automatically request the permissions they need - STS? Check out instance profiles. This feature allows any AWS API-aware application to request credentials on demand, eliminating key management/rotation: docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html
cachemiss 11y ago
So I'd modify these a bit. We run a very large AWS infrastructure as a engineering team (no dedicated ops). 1. Use CloudFormation only for infrastructure that largely doesn't change. Like VPC's, subnets/ internet gateways etc. Do not use it for your instances / databases etc, I can't recommend that enough, you'll get into a place where updating them is risky. We have a regional migration (like database migrations) that runs in each region we deploy to that sets up ASG, RDS etc. It allows us control over how things change. If we need to change a launch conf etc. 2. Use auto-scaling groups in your stateless front ends that don't have really bursty loads, it isn't responsive enough for really sharp spikes (though not much is). Otherwise do your own cluster management if you can (though you should probably default to autoscaling if you can't make a strong case not to use it). 3. Use different accounts for dev / qa / prod etc. Not just different regions. Force yourself to put in the correct automation to bootstrap yourself into a new account / region (we run in 5 regions in prod, and 3 in qa, and having automation is a lifesaver). 4. Don't use ip addresses for things if you can help it, just create a private hosted zone in Route53 and map it that way. 5. Use instance roles, and in dev force devs to put their credentials in a place where they get picked up by the provider chain, don't get into a place where you are copying creds everywhere, assume they'll get picked up from the environment. 6. Don't use DynamoDB (or any non-relational store) until oyu have to (even though it is great), RDS is a great service and you should stick with it as long as you can (you can make it scale a long way with the correct architecture and bumping instance sizes is easy). IMO a relational store is more flexible than others since you (at least with postgres) get transactional guarantees on DDL operations, so it makes it easier to build in correct migration logic. 6. If you are using cloudformation, use troposphere: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere 7. Understand what instances need internet access and which ones don't, so you can either give them public ips, or put in a NAT. Sometimes security teams get grumpy (for good reason) when you open up machines that don't need to be to the internet, even if its just outbound. 8. Set up ELB logging, and pay attention to CloudTrail. 9. We use Cloudwatch Logs, it has its warts (and its a bit expensive), but it's better than a lot of the infrastructure you see out there (we don't generally index our logs, we just need them to be able to be viewed in a browser and exported for grep). It's also easy to get started with, just make sure your date formats are correct. 10. By default, stripe yourself across AZs if possible (and its almost always possible). Don't leave it for later, take the pain up front, you'll be happy about it later. 11. Don't try and be multi-region if you can at first, just replicate your infrastructure into different regions (other than users / accounts etc.). People get hung up on being able to flip back and forth between regions, and its usually not necessary. edit: Track everything in cloudwatch, everything.
prioritydirect 11y ago
Xmas Message From Ghana http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shares-touching.html http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shar...
prioritydirect 11y ago
Xmas Message from Ghana http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shares-touching.html http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shar...
narsil 11y ago
> There is no reason - beside manually managed infrastructure - to not decrease the instance size (number of machines or c3.xlarge to c3.large) if you realize that your EC2 instances are underutilized. CPU/Memory aren't the only measures of underutilization. If you require high instantaneous bandwidth throughput, then the networking capacity available to your instance roughly increases with the size of your instance. This includes both EBS as well as other Network traffic. Table with Low/Medium/High: https://aws.amazon.com/ec2/instance-types/#instance-type-matrix https://aws.amazon.com/ec2/instance-types/#instance-type-mat... Example benchmark with c3 instances: http://blog.flux7.com/blogs/benchmarks/benchmarking-network-performance-analysis-of-c3-instances-using-iperf-tool http://blog.flux7.com/blogs/benchmarks/benchmarking-network-... If you're more concerned with just EBS network throughput, check out the table on this page instead: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-...
- thebigjc 11y ago
  This is super important - I've run a number of 'oversized' instances just to get better networking. AFAIK none of the cloud providers offer a way to create a 'network' optimized instance, though containerized deployments should help with under utilizing the rest of the instance.
  - zbjornson 11y ago
    This was a big issue for me too. Google Compute Engine seems to have no throttling like AWS does, or at least it's a much higher cap. On an n1-standard-1 I routinely get 250+ MB/s, and near 1 GB/s for n1-standard-4 and larger. The only high bandwidth AWS instances I've found are the few expensive ones that spec 10 Gbit.
    - vgt 11y ago
      According to this page, you can get ~14Gbit networking on Google Cloud's n1-standard-8 http://googlecloudplatform.blogspot.com/2015/11/bringing-you-more-flexibility-and-better-Cloud-Networking-performance-GA-of-HTTPS-Load-Balancing-and-Akamai-joins-CDN-Interconnect.html http://googlecloudplatform.blogspot.com/2015/11/bringing-you...
tomglindmeier 11y ago
I wonder if AWS is a good location to run a VoIP server. VoIP is a real time application that is very prone to jitter, latency and packet loss. I'm concerned about "noisy neighbors" and decreased network performance at AWS. Does anybody have experience with running a VoIP (e. g. Asterisk) on AWS?
- discodave 11y ago
  If you're willing to pay, then you can avoid the noisy neighbor problem almost completely by using c3.8xlarge or c4.10xlarge instances. Those instances get their own dedicated 10G NIC. You can use reserved instances to reduce the cost. But even with smaller instances, AWS network performance is going to be about the same as any VPS provider. The only way to get guaranteed performance is to do Colo, which is expensive.
- sciurus 11y ago
  Twilio runs on AWS.
- emerongi 11y ago
  I sell an application built around VoIP. I dislike running my own servers, so just lately I moved them to AWS. At first, I used instances that were not very performant, causing some problems. I quickly moved to more powerful instances and since then there have been no problems at all. There is a slight delay, which is normal, since the servers are not in the same country as the users anymore, but the users haven't noticed it at all. I can't say how well it scales, though. I have a small user-base, so scaling has not been a problem yet. Network performance might become a problem if you have a huge user-base.
  - ant6n 11y ago
    How can you be sure that your users haven't noticed?
- zonywhoop 11y ago
  Check out voicehub.com, their entire stack is on ec2.
nikolay 11y ago
As there's too much push to Terraform, which I personally dislike due to many opinionated features and the marketing push to cover as many services as possible and not do one thing and do it great (AWS), you can look at Bazaarvoice's CloudFormation Ruby DSL [0]. [0]: https://github.com/bazaarvoice/cloudformation-ruby-dsl https://github.com/bazaarvoice/cloudformation-ruby-dsl
sergiotapia 11y ago
I wonder when Amazon is going to invest their ungodly earnings into UX. Their UI is absolutely terrible and I shudder every time I have to log into it. Is it purposely built that way to confuse people who don't belong in there?
- mattbillenstein 11y ago
  I've heard this a few times -- AWS is a very complex product that gives you a lot of freedom in how you build your product using it. I think the web UI works about as well as it could given that complexity. If you want something simpler, you have Heroku and a bunch of similar things which make a bunch of decisions for you -- but you don't have the flexibility there that you do with AWS of course.
  - sergiotapia 11y ago
    There has to be some middle ground between AWS and Heroku though.
    - mattbillenstein 11y ago
      There are some options, but the point is, what you're trading for simpler UX is a simpler overall product with less flexibility.
    - vemv 11y ago
      It's called Elastic Beanstalk! As it name suggests you can start as easily as with Heroku, and adapt it later with whatever additional AWS infrastructure you may need.
ddevault 11y ago
Full disclosure: I work for Linode and I am super biased. This is my own opinion etc etc Perhaps the first AWS mistake you might make is... using AWS? Even before I started at Linode, I thought it was terrible. It's extremely, unreasonably pricey. The UI is terrible. Their offerings are availble elsewhere. I started MediaCrush, a now-defunct media hosting website, on AWS. After a while, we switched to dedicated hosting (really sexy servers). We were looking at $250 a month and scaled up to millions of visitors per day! I ran our bandwidth and CPU usage and such numbers through the AWS price calculator a while ago - over $20,000 per month. AWS is a racket. It seems to me like the easiest way to burn through your new startup's seed money real fast. Edit: not trying to sell you on Linode, just disclosing that I work there. There are lots of options, just do the research before you reach for AWS.
- JabavuAdams 11y ago
  As an AMZN shareholder, I encourage you to please use AWS. That is all.
- tshtf 11y ago
  In the past during security incidents (and there have been more three than major ones in as many years), there has been absolutely no transparency from Linode management. That's more than enough reason for serious customers to avoid Linode...
  - Sir_Cmpwn 11y ago
    I don't think it's appropriate for me to comment on security incidents (before my time, anyway), but I'm not trying to sell people on Linode here. MediaCrush was actually using a combination of Voxility and Digital Ocean (and Route 53, actually). There are lots of options, guys.
  - seany 11y ago
    I was going to say the same thing. The amount of transparency in general from aws is DRAMATICALLY better than linode.
- CharlesW 11y ago
benmanns 11y ago
Has anyone tried the Trusted Advisor feature out? Have you found it worth the 3-10% on top of existing monthly usage?
- wsh91 11y ago
  If you already have a support plan, it's free. So yes, I have. Nothing useful so far, but I plan to keep checking it periodically in case I'm off my game.
matdrewin 11y ago
Personally never quite got the appeal for EC2. You're basically replicating what you would be doing on physical servers anyway. The real productivity gains come from using a fully managed PaaS (Heroku, Azure Web Apps, Google App Engine, OpenShift etc.) where there is no maintenance and scaling and redundancy is taken care for you. Granted those are even more expensive but they are the only ones that provide any kind of value.
overgard 11y ago
This might be the wrong place to ask, but I'm curious how people feel Azure stacks up to AWS? The services seem comparable (maybe even nicer), but I'm unclear how it compares on cost.
ap22213 11y ago
I don't get all the complaints about AWS prices. I just spun up a 60 node Spark cluster with over 5TB of memory, processed 100B data records, and spent $5.50!
fibo 11y ago
Nice article, I am working in a small company (Beintoo) dice september 2015 and we use AWS here. I think is a very interesting set if products and you can build any kind of business. For sure the advices given in the article are really useful, in fact I will apply them at my job place. About comparison with other services, I was working in Deloitte Analytics before, managing the cloud services provided by IBM Softlayer. You cannot compare them, AWS offers many more and I was not really satisfied with SoftLayer, for example I had a problem with a network upgrade they did on January 2014 and I have lost a lot of data, with poor support to restore it. Also the starting price of 25$ per month is really expensive. AWS is far more mature and interesting. Then for my own servers I use CloudAtCost cause is cheaper but if I run a business for sure I would go with AWS. If you gain money, is not that expensive and if you stick with Amazon advices and philosophy is very reliable.
llamas02 11y ago
Thinking the wrong thing and feelings getting hurt. Picking the person who doesn't pick you. Forgetting someone and making them feel unimportant. Not choosing your kids firstly and when they don't choose you and your next choice is your biggest mistake. Getting lost in love that doesn't exist.letting media control you. Listening to rumors. Bad communication, and discernment. Letting anyone control you or take your rights away. Running away. Illegally being blocked by someone using our services and it effecting my pursuit of happiness. Letting go of discrimination against me, and not pursuing Justice for myself and family and friends. Not checking and fixing paperwork and getting involved in govt. Fraud. From banks, public agencies and law. Not fighting for my rights as a human being and continuing to let these assholes invade my privacy. Letting others steal my ideas and my life. Letting the law say I'm guilty before innocent. Having instincts and not reacting fast enough. Letting others talk bad to me without hitting them.Not tracking my money. Writing my brother. Not getting a career as a secret shopper and shutting down all bad utility companies for fraud and bad businesses. Letting others control my thoughts. Ever admitting I was an addict. Getting excited over nothing. Being too humble that it weakens you. Not accepting I'm crazy. Feeling sorry for everyone but myself. Helping everyone over ME. Letting my son leave. Not cleaning the other room so there's more room. Waiting on someone to save you when all you have is yourself. Believing in dreams come true.
CRConrad 11y ago
From the article: "The biggest problem with Auto Scaling Groups is that people assume that they are about auto scaling which they are not!" Hmm... Then maybe the biggest problem with them is actually that they are misnamed?
vitoc 11y ago
I work in a team that uses quite a lot of AWS for various reasons beyond cost. We also use various other clouds too, depending on needs of a specific project or situation. Our experience with cloud is that it is a journey. Obviously, coupled with DevOps tooling, it’d allowed us to deal with environment requirements at the speed of software, i.e. Infrastructure as Code and all. One thing we find ourselves doing over and over is changing the infrastructure, either because requirements change, new cloud services are launched and its useful to us or simply because we find more efficient ways of running cloud resources (Trusted Advisor helps). We’d built a tool called Liquid Sky (https://liquidsky.singtel-labs.com https://liquidsky.singtel-labs.com) to help us keep track of the cost impact of the changes we make constantly. I did mention that we use cloud for reasons beyond cost, but we definitely still want to know that we’re sensible and maximise cost efficiency as well, its just another (important) factor. Because we change our cloud resources so frequently, we didn’t want to make it a very rigid process when dealing with the sensibilities of cloud cost. Hence, we’d built Liquid Sky in a way that gives our engineers the freedom to explore better way of running things on the cloud while keeping cost in check as well as keeping the team (including cost guardians) in the loop.

537 points by hellomichibye 11y ago

Also, dont use Dynamo unless you have a really good reason. "It scales better than MySQL" is not a good reason. Have fun migrating data and re-indexing constantly!

chucky_z 11y ago
What about as a kind of Redis stand-in? I'm curious about the DynamoDB PHP session driver. It sounds like you have a lot of experience with Dynamo. All the use-cases I seem to keep coming up with are more for storing global environment keys outside of the environment itself and using a large number of IAM roles to access individual keys, and as a kind of throwaway 'I need to store this somewhere, but it doesn't really fit in the main DB, and I still need it to persist for at least awhile' case.
- kelnos 11y ago
  Not the grandparent, but I have a lot of experience with DynamoDB as well. I migrated a self-managed, sharded redis store to DDB close to a year ago, and what it really depends on are your read and write patterns, and whether or not you can live with the opacity of what DDB is doing behind the scenes. An example: if you provision X capacity units, you're actually provisioning X/N capacity units per partition. AWS is mostly transparent (via documentation) about when a table splits into a new partition (I say "mostly" because I was told by AWS that the numbers in the docs aren't quite right), but you'll have no idea how the keys are hashed between partitions, and you won't know if you have a hot partition that's getting slammed because your keys aren't well-distributed. Well, no, I take that back -- you will know, because you'll get throttled even though you're not consuming anywhere near the capacity you've provisioned. You just won't know how to fix it without a lot of trial and error (which isn't great in a production system). If your r/w usage isn't consistent, you'll either have periods of throttling, or you'll have to over-provision and waste money. There's no auto-scaling like you can do with EC2. Not trying to knock the product: still using DDB... but getting it to a point where I felt reasonably confident about it took way longer than managing my own data store... and then they had a 6-hour outage one early, early Sunday morning a couple months ago. Possibly solution: dual-writes to two AWS regions at once and the ability to auto-pivot reads to the backup region. Which of course doubles provisioning costs. Ok, maybe I'm knocking it a little. It's a good product, but there are definitely tradeoffs.
  - chucky_z 11y ago
    Was the ElastiCache service around when you migrated Redis to Dynamo? This is another alternative I'm considering.
    - kelnos 11y ago
      The problem with ElastiCache -- and why I rejected it as an option -- as that they make you define a 30-minute "maintenance window" where AWS can apply patches and reboot your cache instances. In practice I've heard that this happens rarely, and when it does, the downtime is short, but it can in theory cause longer outages. And in the case of both the redis and memcached backends, if the maintenance requires restarting redis/memcached or rebooting the instance, you lose all data in the cache (at least up until your last backup). For this particular project, that amount of downtime would easily cause a real outage for customers, and was unacceptable.
  - boomzilla 11y ago
    You should put some effort on designing the key schema, especially the hash key. Don't do timestamp, or sequence. I had to do a 'encrypted' key schema on my sequence IDs so sequential records are spread into different partitions. Actually it worked out well as I can exposed these encrypted keys externally too. Agreed that DDB documentation could be better on the best practices. I found this deep dive youtube video very helpful: https://www.youtube.com/watch?v=VuKu23oZp9Q https://www.youtube.com/watch?v=VuKu23oZp9Q
    - kelnos 11y ago
      Oh, absolutely. The problem is that your key schema also ties you to what kind of queries you can do efficiently. My primary keys actually are very well-distributed across the hash space, but in my initial implementation, one of the global secondary indexes had a key that caused throttling and absolutely destroyed performance. Dropping that index meant losing the ability to do an entire class of queries. In this case, we were (sorta) able to live with that, but I can imagine many cases where that's a problem. That's actually another instance of the lack of transparency and trial-and-error: "oh hey, writes are getting throttled... no idea why... let's drop this index and see if it helps".
markism 11y ago
Beginner here, how do you keep the database available to multiple instances without Dynamo?
- jdimov9 11y ago
  Use RDS.
dexterdog 11y ago
Can you expand on your last statement?
greenleafjacob 11y ago
Is local secondary indices and lower operational costs than Cassandra a good reason(s)?
- Rapzid 11y ago
  Not without considering the throttling you will hit if you exceed provisioned IOPS. You really want to consider all the angles on dynamoDB before committing to it; limitations, unique benefits, interface, etc.
- wsh91 11y ago
  Operational costs are in the eye of the beholder. We evaluated DynamoDB for a new use case but we're going with Cassandra because the storage costs alone more than made up for ops overhead. At least IMO, Cassandra's support for multicolumn range queries (via clustering columns) and the cheaper storage you can use obviate local secondary indices. (Cassandra has secondary indices as well but it seems like most folks prefer further denormalization--at least, that's what we're doing.)
joshuahornby 11y ago
Can you explain these comments? I am currently developing a system using Dyanmo as the database, I need a JSON like structure as the documents need to be flexible as different documents will contain different keys. So I am interested to hear why you think Dyanmo is a bad choice.
ap22213 11y ago
I use Dynamo because I'm lazy. Need some JSON fast? 3 lines and no worries. DynamoDB db = new DynamoDB(new AmazonDynamoDBClient()); Table myTable = db.getTable("table"); Item myItem = myTable.getItem("id", id); Sure, I don't use it for a lot of things, but for cross-VPC / cross-account configuration, it's pretty nice.

andrioni 11y ago

I second the recommendation to use CloudFormation + packer + ELBs + auto scaling groups for web applications whenever possible, it just makes everything so easy and automatic. Of course, there's a learning curve and you pay a premium for all that automation, but in my experience it has been usually worth it so far.

ZacharyPitts 11y ago
I like going one step further and making deployments and AWS resources into reusable code using Troposphere: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere

deleted 11y ago

[deleted]

hellomichibye 11y ago
+1 I concentrated on non security related mistakes. security will follow next week... :)
startitrrr3 11y ago
Great points siddharth_mal

kevindeasis 11y ago

You know what I've realized that's really important. More AWS tutorials is really needed. There's numerous of new programmers who want to learn AWS, but can't finish building anything because they get buried in documentation. I find there are a lot of high-level abstracted tutorials, but for the new services, there aren't a lot of detailed tutorials. For instance, an implemented cognito->gateway->lambda->dynamodb is really hard for a newbie to do.

ninjay 11y ago
I found this series very helpful: https://medium.com/aws-activate-startup-blog https://medium.com/aws-activate-startup-blog
jcyw 11y ago
I have some good experience with qwiklab. Their self-paced lab starts with UI, then turned to aws cli. They also give a nice lesson on CF.
euphemize 11y ago
Great point. Also, the pipeline you mention removes many risks related to the 'common' mistakes mentioned in the article (scaling, monitoring, provisioning it all done for you - or almost). We're using gateway -> lambda -> dynamodb, and there are a tons of gotchas and small things that AWS need to iron out, especially with gateway -> lambda.
- LoSboccacc 11y ago
  Agreed. Was looking into that for a serverless model inspired by (1) but there is still loads of things they can't do/won't do 1 https://github.com/serverless/serverless https://github.com/serverless/serverless
  - ac360 11y ago
    Working on it :) We are hacking away full time on the Serverless Framework and have some big features coming in the next few days.

nodesocket 11y ago

My huge recommendation is to put production instances in a completely seperate region than development and staging instances. I actually just discovered that you can limit IAM API keys to a specific region, you just need to create a custom policy. The following policy is an example: { "Version": "2012-10-17", "Statement": [ { "Sid": "SOME-ID-HERE", "Effect": "Allow", "Action": [ "ec2:*" ], "Condition": { "StringEquals": { "ec2:Region": "us-west-2" } }, "Resource": [ "*" ] } ] }

cheetos 11y ago
Why?
- nodesocket 11y ago
  If you segment production instances from develop/staging you can use IAM rules to grant specific privileges based on the entire region, instead of by tag (which is fragile). Additionally, it is less error prone when you are making manual changes in the console as it requires switching regions between production and develop/staging.
eropple 11y ago
Why would you do this instead of using multiple AWS accounts? Different regions have different feature sets (available instance types, beta eligibility, etc.). I strongly recommend instead using multiple AWS accounts instead and keeping them in the same region. That said, since you should be using an infrastructure provisioning tool like CloudFormation, the tagging solution should not be a particularly big obstacle.
- kbar13 11y ago
  yes, use multiple accounts. you can use STS to grant permissions between the accounts if needed.

ninjay 11y ago

What are the current ways to make creating CloudFormation templates not so painful?

loki77 11y ago
You should check out troposphere (https://github.com/cloudtools/troposphere/ https://github.com/cloudtools/troposphere/) and stacker (https://github.com/remind101/stacker/ https://github.com/remind101/stacker/). I'm a maintainer of both - we try to make CF easier by catching errors earlier, and allowing you to do things like write for loops, in troposphere. stacker tries to take your troposphere templates and tie them together as totally separate stacks.

siddharth_mal 11y ago

I'd like to add two more: 1) Not giving out your access and secret keys in scripts/buckets. 2) Always using IAM roles with your EC2

Bestcoderplanet 11y ago
+1 I concentrated on non security related mistakes. security will follow next week... :)
misiti3780 11y ago
how do you avoid 1 - it seems impossible ?
- mhluongo 11y ago
  IAM roles let you assign temporary credentials to machines running scripts. The machine can then hit an internal AWS URL to get the temporary credentials. Many tools know to look for these credentials by default- eg boto checks for credentials in environment variables, config files, and the machines IAM role.
  - idunno246 11y ago
    And there's a few tools to emulate the metadata service locally if you need it on dev laptops which makes it use a role as if a server
- rodrickbrown 11y ago
  Take a look at hashicorp vault - https://hashicorp.com/blog/vault.html https://hashicorp.com/blog/vault.html
hrez 11y ago
IAM roles is ok as long as you realize that anything and anybody on that instance gets access to those credentials.

peterwaller 11y ago

I like CloudFormation. Unfortunately it is very unwieldy to write CloudFormation templates directly, and we're not about to start using the AWS CFN GUI editor! It seems like the assembly of the AWS ecosystem. Does anyone else have a favourite hammer for this particular nail? I'd love to have something better than our home-baked solution, but I'm yet to find anything which doesn't introduce other flaws, such as an incomplete implementation (missing parameters or resource types) or ultimately making a leaky abstraction on top of CloudFormation somehow. I ended up brewing a reasonably straightforward solution using Python as a (minimal) DSL which emits JSON. Its primary purpose is to support the whole of the CFN ecosystem (not just implement some small part of EC2, for instance) while also not trying to be too clever. It has about 50-100 lines of python which implements helper functions such as ref(), join() and load_user_data(), and not many other things. There is an almost 1-to-1 correspondence between the generated CFN configuration and the python source. As a bonus it checks for a few common mistakes like broken refs or parameters which aren't used. I have heard that similar solutions have been reinvented in a few places, including the BBC. But I'm yet to see a good public solution!

otterley 11y ago
Some people like SparkleFormation (http://www.sparkleformation.io/ http://www.sparkleformation.io/). I'll warn you, though, that it's not a good example of how to program in Ruby. It abuses method_missing to the point that it makes your implementations difficult to debug.
- nikolay 11y ago
  I also don't like their made-up terms such as "dynamics", etc. The documentation is pretty confusing as well.
andystanton 11y ago
Troposphere (https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere) is a mature Python CloudFormation solution that sounds similar to your home brewed one.

kennu 11y ago

I warmly recommend the Serverless framework for building basic web applications on AWS. It handles CloudFormation details for you, but lets you customize them if needed. Not suitable for every possible app though.

rgawdzik 11y ago

Check out http://convox.com/ http://convox.com/ (YC S15) for an alternative to avoiding manual infrastructure.

nzoschke 11y ago
Convox co-founder here. Thanks for the shout out! That's exactly right, avoid manual infrastructure. Someday we will all have something like Rails for infrastructure. Strong conventions around best practices. If you follow these conventions you can avoid bespoke or manual configurations and focus solely on your app logic. We're building Convox to advance this goal.

misiti3780 11y ago

Another, somewhat obvious one: Be extremely careful when using public customized AMIs, a lot of times ~/.ssh/authorized_hosts contains public keys and this is obviously a huge security problem

kbar13 11y ago
no, public keys are not a security problem.
- joseph 11y ago
  They are if you don't know who has the corresponding private key.
  - kbar13 11y ago
    oh right when using public AMIs. I thought parent meant publishing AMIs. derp.
- iancarroll 11y ago
  But the fact they allow access to your system is.
- mhurron 11y ago
  Public keys on your host where you don't control who has the private key are.
- deleted 11y ago
  [deleted]
nodesocket 11y ago
Yeah, I almost feel like AWS should clear the contents of authoried_keys for each user when you make an AMI public. That of course is bound to break some things, but prevents security oversights.
- brianwawok 11y ago

T3RMINATED 11y ago

By choosing AWS to begin with you have commited the Top 1 Mistake that will come to haunt you.

buremba 11y ago

Use OpsWorks if possible. It's free and provides a simple interface that allows you to deploy/upgrade your apps automatically and monitors your instances automatically using CloudWatch.

wsh91 11y ago
+1. AWS has done an amazing job with OpsWorks. It's free Chef 12. What's not to love? (The only drawback I've noticed so far is some standard Chef stuff--vault in particular--not working. Other than that, very pleased.)

diziet 11y ago

Something major is missing: Running Demand instances instead of Reserved

hrez 11y ago
Or running Reserved instead of onDemand. It's all about use cases.

tedmiston 11y ago

I'll add one that was especially common for people coming off the year of free tier a couple years ago. I'm not sure that AWS has changed it yet. 6. Not starting a box/instance/database and forgetting it's running until you receive the bill after your free tier expires.

molecule 11y ago
I expected to see something like this generalized in the article as 6. Not setting up billing alarms, http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...

ommunist 11y ago

I attended aws workshop, several years ago, there I realized, that this vast ecosystem of infrastructure services require a popularisation effort of similar scale. And not just in plain English. IAM that days was not very clearly understandable, search supported only ascii and was not really documented, now that ecosystem is in order of magnitude larger and more complex. And efforts like cloudonaut's should be greatly appreciated. For the greater public good. Thank you, man!

dantiberian 11y ago

The biggest mistake I've seen with AWS (and committed myself), is not reading the manuals for the services you're using. While some people complain about the AWS manuals not being complete, there is still a lot of good information in there that you might miss if you're just clicking through the console.

elwell 11y ago

I'd like to add, if using Elastic Beanstalk, don't directly attach an RDS instance when creating the environment. If you do, you won't be able to destroy your environment without also deleting the RDS instance. Instead, create the RDS instance separately, and just add the proper security group for the environment to be able to access the host. Then you can easily create a new eb environment with any config changes (there are some config changes you can't make to an eb environment without creating a new one from scratch) and then connect to your existing db.

gwintrob 11y ago
Is there a site that collects these sorts of tips? Seems like there are a lot of potential pitfalls that can be hard to find in docs like: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-rds.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-r...
softdev12 11y ago
interesting. i was under the impression you could just create a snapshot (before destroying the environment) and then restore as needed. am i missing something? Edit: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R...
- sdroy 11y ago
  If you can take the database instance offline and if you do not need to access it from outside of the beanstalk app then that is fine.

vacri 11y ago

> There is no reason why you should manage your infrastructure manually. It's unprofessional! It's a mess! Nonsense. Cloudformation has it's issues. It takes time to learn and implement. The templates can break, requiring the stack to be destroyed and remade. In the sample in the article, the database is in the same template as everything else - what fun that will be when an update breaks the template and you have to reapply the stack (which destroys the existing database). Cloudformation is good, but it comes with caveats, and the idea that you should only manage an AWS stack with CF is utter tripe. As with everything, it depends on your use case. Also weird is the article's demand of using autoscaling groups to monitor single instances. Why not just monitor them directly with cloudwatch? > There is no reason - beside manually managed infrastructure - to not decrease the instance size (number of machines or c3.xlarge to c3.large) if you realize that your EC2 instances are underutilized. This is wrong, too. Autoscaling takes time to scale up, and it scales up in stages. If you get sudden traffic, autoscaling can take too long. Again, it's about knowing your use case. Unfortunately for us, we can get sudden traffic when one of our clients does a media release and they don't tell us ahead of time, The five or so minutes it takes for instances to trigger the warning, start up a new set, and then attach these to a load balancer is too long for this particular use case, so we just have to run with a certain amount of excess capacity. Autoscaling is awesome, but this article is way too didactic in it's No True Scotsman approach.

Rapzid 11y ago
Cloudformation can really be a harsh mistress. I feel I'm constantly discovering good reasons for not including certain sets of resources in the same templates for reasons you and others touch on. And nested stacks? You know, never say never.. Probably never again.

voltagex_ 11y ago

I find IAM particularly difficult to use - I feel like there should be a button to create a user/group that can do only X, Y, Z. I realise policy templates get most of the way there but I still had to go and read the syntax for them because DescribeRegions wasn't in the list I needed. I'm also not sure how to make the jump from exporting AWS_ACCESS_KEY_ID and having my instances automatically request the permissions they need - STS?

inopinatus 11y ago
If your code is using the AWS SDK then you don't need to export an access key for your running code. Just create an EC2 Service Role in IAM with the policy you want attached to the role; then launch the instance with that role, to get automatic temporary credentials. Refs: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use... http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles...
cheeseprocedure 11y ago
> I'm also not sure how to make the jump from exporting AWS_ACCESS_KEY_ID and having my instances automatically request the permissions they need - STS? Check out instance profiles. This feature allows any AWS API-aware application to request credentials on demand, eliminating key management/rotation: docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html

cachemiss 11y ago

So I'd modify these a bit. We run a very large AWS infrastructure as a engineering team (no dedicated ops). 1. Use CloudFormation only for infrastructure that largely doesn't change. Like VPC's, subnets/ internet gateways etc. Do not use it for your instances / databases etc, I can't recommend that enough, you'll get into a place where updating them is risky. We have a regional migration (like database migrations) that runs in each region we deploy to that sets up ASG, RDS etc. It allows us control over how things change. If we need to change a launch conf etc. 2. Use auto-scaling groups in your stateless front ends that don't have really bursty loads, it isn't responsive enough for really sharp spikes (though not much is). Otherwise do your own cluster management if you can (though you should probably default to autoscaling if you can't make a strong case not to use it). 3. Use different accounts for dev / qa / prod etc. Not just different regions. Force yourself to put in the correct automation to bootstrap yourself into a new account / region (we run in 5 regions in prod, and 3 in qa, and having automation is a lifesaver). 4. Don't use ip addresses for things if you can help it, just create a private hosted zone in Route53 and map it that way. 5. Use instance roles, and in dev force devs to put their credentials in a place where they get picked up by the provider chain, don't get into a place where you are copying creds everywhere, assume they'll get picked up from the environment. 6. Don't use DynamoDB (or any non-relational store) until oyu have to (even though it is great), RDS is a great service and you should stick with it as long as you can (you can make it scale a long way with the correct architecture and bumping instance sizes is easy). IMO a relational store is more flexible than others since you (at least with postgres) get transactional guarantees on DDL operations, so it makes it easier to build in correct migration logic. 6. If you are using cloudformation, use troposphere: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere 7. Understand what instances need internet access and which ones don't, so you can either give them public ips, or put in a NAT. Sometimes security teams get grumpy (for good reason) when you open up machines that don't need to be to the internet, even if its just outbound. 8. Set up ELB logging, and pay attention to CloudTrail. 9. We use Cloudwatch Logs, it has its warts (and its a bit expensive), but it's better than a lot of the infrastructure you see out there (we don't generally index our logs, we just need them to be able to be viewed in a browser and exported for grep). It's also easy to get started with, just make sure your date formats are correct. 10. By default, stripe yourself across AZs if possible (and its almost always possible). Don't leave it for later, take the pain up front, you'll be happy about it later. 11. Don't try and be multi-region if you can at first, just replicate your infrastructure into different regions (other than users / accounts etc.). People get hung up on being able to flip back and forth between regions, and its usually not necessary. edit: Track everything in cloudwatch, everything.

prioritydirect 11y ago

Xmas Message From Ghana http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shares-touching.html http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shar...

prioritydirect 11y ago

Xmas Message from Ghana http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shares-touching.html http://nanabkay.blogspot.com/2015/12/pretty-esi-dorinda-shar...

narsil 11y ago

> There is no reason - beside manually managed infrastructure - to not decrease the instance size (number of machines or c3.xlarge to c3.large) if you realize that your EC2 instances are underutilized. CPU/Memory aren't the only measures of underutilization. If you require high instantaneous bandwidth throughput, then the networking capacity available to your instance roughly increases with the size of your instance. This includes both EBS as well as other Network traffic. Table with Low/Medium/High: https://aws.amazon.com/ec2/instance-types/#instance-type-matrix https://aws.amazon.com/ec2/instance-types/#instance-type-mat... Example benchmark with c3 instances: http://blog.flux7.com/blogs/benchmarks/benchmarking-network-performance-analysis-of-c3-instances-using-iperf-tool http://blog.flux7.com/blogs/benchmarks/benchmarking-network-... If you're more concerned with just EBS network throughput, check out the table on this page instead: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-...

thebigjc 11y ago
This is super important - I've run a number of 'oversized' instances just to get better networking. AFAIK none of the cloud providers offer a way to create a 'network' optimized instance, though containerized deployments should help with under utilizing the rest of the instance.
- zbjornson 11y ago
  This was a big issue for me too. Google Compute Engine seems to have no throttling like AWS does, or at least it's a much higher cap. On an n1-standard-1 I routinely get 250+ MB/s, and near 1 GB/s for n1-standard-4 and larger. The only high bandwidth AWS instances I've found are the few expensive ones that spec 10 Gbit.
  - vgt 11y ago
    According to this page, you can get ~14Gbit networking on Google Cloud's n1-standard-8 http://googlecloudplatform.blogspot.com/2015/11/bringing-you-more-flexibility-and-better-Cloud-Networking-performance-GA-of-HTTPS-Load-Balancing-and-Akamai-joins-CDN-Interconnect.html http://googlecloudplatform.blogspot.com/2015/11/bringing-you...

tomglindmeier 11y ago

I wonder if AWS is a good location to run a VoIP server. VoIP is a real time application that is very prone to jitter, latency and packet loss. I'm concerned about "noisy neighbors" and decreased network performance at AWS. Does anybody have experience with running a VoIP (e. g. Asterisk) on AWS?

discodave 11y ago
If you're willing to pay, then you can avoid the noisy neighbor problem almost completely by using c3.8xlarge or c4.10xlarge instances. Those instances get their own dedicated 10G NIC. You can use reserved instances to reduce the cost. But even with smaller instances, AWS network performance is going to be about the same as any VPS provider. The only way to get guaranteed performance is to do Colo, which is expensive.
sciurus 11y ago
Twilio runs on AWS.
emerongi 11y ago
I sell an application built around VoIP. I dislike running my own servers, so just lately I moved them to AWS. At first, I used instances that were not very performant, causing some problems. I quickly moved to more powerful instances and since then there have been no problems at all. There is a slight delay, which is normal, since the servers are not in the same country as the users anymore, but the users haven't noticed it at all. I can't say how well it scales, though. I have a small user-base, so scaling has not been a problem yet. Network performance might become a problem if you have a huge user-base.
- ant6n 11y ago
  How can you be sure that your users haven't noticed?
zonywhoop 11y ago
Check out voicehub.com, their entire stack is on ec2.

nikolay 11y ago

As there's too much push to Terraform, which I personally dislike due to many opinionated features and the marketing push to cover as many services as possible and not do one thing and do it great (AWS), you can look at Bazaarvoice's CloudFormation Ruby DSL [0]. [0]: https://github.com/bazaarvoice/cloudformation-ruby-dsl https://github.com/bazaarvoice/cloudformation-ruby-dsl

sergiotapia 11y ago

I wonder when Amazon is going to invest their ungodly earnings into UX. Their UI is absolutely terrible and I shudder every time I have to log into it. Is it purposely built that way to confuse people who don't belong in there?

mattbillenstein 11y ago
I've heard this a few times -- AWS is a very complex product that gives you a lot of freedom in how you build your product using it. I think the web UI works about as well as it could given that complexity. If you want something simpler, you have Heroku and a bunch of similar things which make a bunch of decisions for you -- but you don't have the flexibility there that you do with AWS of course.
- sergiotapia 11y ago
  There has to be some middle ground between AWS and Heroku though.
  - mattbillenstein 11y ago
    There are some options, but the point is, what you're trading for simpler UX is a simpler overall product with less flexibility.
  - vemv 11y ago
    It's called Elastic Beanstalk! As it name suggests you can start as easily as with Heroku, and adapt it later with whatever additional AWS infrastructure you may need.

ddevault 11y ago

Full disclosure: I work for Linode and I am super biased. This is my own opinion etc etc Perhaps the first AWS mistake you might make is... using AWS? Even before I started at Linode, I thought it was terrible. It's extremely, unreasonably pricey. The UI is terrible. Their offerings are availble elsewhere. I started MediaCrush, a now-defunct media hosting website, on AWS. After a while, we switched to dedicated hosting (really sexy servers). We were looking at $250 a month and scaled up to millions of visitors per day! I ran our bandwidth and CPU usage and such numbers through the AWS price calculator a while ago - over $20,000 per month. AWS is a racket. It seems to me like the easiest way to burn through your new startup's seed money real fast. Edit: not trying to sell you on Linode, just disclosing that I work there. There are lots of options, just do the research before you reach for AWS.

JabavuAdams 11y ago
As an AMZN shareholder, I encourage you to please use AWS. That is all.
tshtf 11y ago
In the past during security incidents (and there have been more three than major ones in as many years), there has been absolutely no transparency from Linode management. That's more than enough reason for serious customers to avoid Linode...
- Sir_Cmpwn 11y ago
  I don't think it's appropriate for me to comment on security incidents (before my time, anyway), but I'm not trying to sell people on Linode here. MediaCrush was actually using a combination of Voxility and Digital Ocean (and Route 53, actually). There are lots of options, guys.
- seany 11y ago
  I was going to say the same thing. The amount of transparency in general from aws is DRAMATICALLY better than linode.
CharlesW 11y ago

benmanns 11y ago

Has anyone tried the Trusted Advisor feature out? Have you found it worth the 3-10% on top of existing monthly usage?

wsh91 11y ago
If you already have a support plan, it's free. So yes, I have. Nothing useful so far, but I plan to keep checking it periodically in case I'm off my game.

matdrewin 11y ago

Personally never quite got the appeal for EC2. You're basically replicating what you would be doing on physical servers anyway. The real productivity gains come from using a fully managed PaaS (Heroku, Azure Web Apps, Google App Engine, OpenShift etc.) where there is no maintenance and scaling and redundancy is taken care for you. Granted those are even more expensive but they are the only ones that provide any kind of value.

overgard 11y ago

This might be the wrong place to ask, but I'm curious how people feel Azure stacks up to AWS? The services seem comparable (maybe even nicer), but I'm unclear how it compares on cost.

ap22213 11y ago

I don't get all the complaints about AWS prices. I just spun up a 60 node Spark cluster with over 5TB of memory, processed 100B data records, and spent $5.50!

fibo 11y ago

Nice article, I am working in a small company (Beintoo) dice september 2015 and we use AWS here. I think is a very interesting set if products and you can build any kind of business. For sure the advices given in the article are really useful, in fact I will apply them at my job place. About comparison with other services, I was working in Deloitte Analytics before, managing the cloud services provided by IBM Softlayer. You cannot compare them, AWS offers many more and I was not really satisfied with SoftLayer, for example I had a problem with a network upgrade they did on January 2014 and I have lost a lot of data, with poor support to restore it. Also the starting price of 25$ per month is really expensive. AWS is far more mature and interesting. Then for my own servers I use CloudAtCost cause is cheaper but if I run a business for sure I would go with AWS. If you gain money, is not that expensive and if you stick with Amazon advices and philosophy is very reliable.

llamas02 11y ago

Thinking the wrong thing and feelings getting hurt. Picking the person who doesn't pick you. Forgetting someone and making them feel unimportant. Not choosing your kids firstly and when they don't choose you and your next choice is your biggest mistake. Getting lost in love that doesn't exist.letting media control you. Listening to rumors. Bad communication, and discernment. Letting anyone control you or take your rights away. Running away. Illegally being blocked by someone using our services and it effecting my pursuit of happiness. Letting go of discrimination against me, and not pursuing Justice for myself and family and friends. Not checking and fixing paperwork and getting involved in govt. Fraud. From banks, public agencies and law. Not fighting for my rights as a human being and continuing to let these assholes invade my privacy. Letting others steal my ideas and my life. Letting the law say I'm guilty before innocent. Having instincts and not reacting fast enough. Letting others talk bad to me without hitting them.Not tracking my money. Writing my brother. Not getting a career as a secret shopper and shutting down all bad utility companies for fraud and bad businesses. Letting others control my thoughts. Ever admitting I was an addict. Getting excited over nothing. Being too humble that it weakens you. Not accepting I'm crazy. Feeling sorry for everyone but myself. Helping everyone over ME. Letting my son leave. Not cleaning the other room so there's more room. Waiting on someone to save you when all you have is yourself. Believing in dreams come true.

CRConrad 11y ago

From the article: "The biggest problem with Auto Scaling Groups is that people assume that they are about auto scaling which they are not!" Hmm... Then maybe the biggest problem with them is actually that they are misnamed?

vitoc 11y ago

I work in a team that uses quite a lot of AWS for various reasons beyond cost. We also use various other clouds too, depending on needs of a specific project or situation. Our experience with cloud is that it is a journey. Obviously, coupled with DevOps tooling, it’d allowed us to deal with environment requirements at the speed of software, i.e. Infrastructure as Code and all. One thing we find ourselves doing over and over is changing the infrastructure, either because requirements change, new cloud services are launched and its useful to us or simply because we find more efficient ways of running cloud resources (Trusted Advisor helps). We’d built a tool called Liquid Sky (https://liquidsky.singtel-labs.com https://liquidsky.singtel-labs.com) to help us keep track of the cost impact of the changes we make constantly. I did mention that we use cloud for reasons beyond cost, but we definitely still want to know that we’re sensible and maximise cost efficiency as well, its just another (important) factor. Because we change our cloud resources so frequently, we didn’t want to make it a very rigid process when dealing with the sensibilities of cloud cost. Hence, we’d built Liquid Sky in a way that gives our engineers the freedom to explore better way of running things on the cloud while keeping cost in check as well as keeping the team (including cost guardians) in the loop.