12 ms·

Using a single AWS account is a serious risk

322 points by hellomichibye 11y ago

azinman2 11y ago
Like the solution via MFA & temp credentials! Simple & elegant.
deleted 11y ago
[deleted]
- marblar 11y ago
  > Why is there a tophat over "IAM role”? Presumably because rather than being a person, the IAM role is a “hat” that a person wears.
impostervt 11y ago
About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. The next day I got an email from Amazon, alerting me to the problem. Apparently, they scrape github looking for just that kind of stupidity. I instantly deleted the project, but it was too late. Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully. I'm now a zealous advocate for making sure a person never even HAS AWS credentials. Instead, make a new user without a password for each use case, and manually select the privileges that account has. If you have a password to AWS, you shouldn't have credentials.
- e40 11y ago
  I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things. What kills me is there is no easy way to stop all the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.
  - pests 11y ago
    You can automate the stopping via the CLI interface. While I don't think there is a single command to stop all instances, you should be able to whip up a script to get all id's then call the stop-instances command on each of them.
  - agumonkey 11y ago
    You must lose a fair amount of water when you're greeted with a surprise bill of $11k.
    - e40 11y ago
      Yeah, it's a rare moment in life. Especially fun since it happened on a Saturday morning (the notification, that is).
  - copperx 11y ago
    When people find the accounts, what do they use them for? Mining bitcoin?
  - mrestko 11y ago
    Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?
    - fpgaminer 11y ago
      It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...
      - icelancer 11y ago
        ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.
- aikah 11y ago
  > Coding fast and dumb, And that's why you should use Bibucket first and foremost. And then when you vet your stuff and you know it's clean, you can publish it on publicly Github. If you rush pushing code on github public repos that's exactly what will happen. If you start with a private one then move to a public one you have more time to think things through. Now one could imagine a third party service warning users of potential issues with files pushed,based on their name/extension/folder name. But privacy first , then open the code to the public.
  - bosdev 11y ago
    You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. This solution is also strange because if you ever committed anything private it will be in the history. So to make this work you either have to rebase over some history, or lose all the history.
    - zippergz 11y ago
      I have a lot of private repos. One for just about every little project I've done. Some larger projects have more than one. It would be pretty expensive to keep all of this in Github that way. The alternative is to combine a bunch of unrelated stuff into fewer repos (or not keep all of my projects in source control), neither of which is especially appealing. I'd be happy to pay Github something for my usage and to support development. I believe in paying people for work that I find useful. But the pricing model makes the cost disproportionate with the value for me.
    - eropple 11y ago
      > You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. Github doesn't have a personal tier big enough for my private repos. (And I'd host my own before paying $50/month.) I do pay Bitbucket, though, because I have a consistent and small-enough group of collaborators that it makes sense.
    - belovedeagle 11y ago
- 0x0 11y ago
  Github provides a public near-realtime stream of events, I'm sure blackhats are constantly monitoring it looking for private keys all the time. Even if you undo a commit within seconds, it may be too late. https://api.github.com/events https://api.github.com/events
  - meddlepal 11y ago
    I did not know that existed and that is pretty awesome!
  - unethical_ban 11y ago
    If Amazon (and my coworker in a 3 person dev shop) can automate the scanning of common API key patterns as a pre-commit filter, I wonder why Github itself doesn't flag for this sort of thing before publishing.
    - giaour 11y ago
      IIRC, a regex that matches a generic AWS access key will also match a git hash.
      - giaour 11y ago
        Not sure what I said that warrants multiple downvotes. Git commits are identified by SHA1 hashes, so they would be caught by the same regex that would catch AWS keys. According to the AWS security blog, this is the regex you should use for secret keys: `(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])`. To match git hashes, you can use this one: `[0-9a-f]{40}`. See the likeness?
    - IanCal 11y ago
      What should they do? Reject a "git push"?
      - geofft 11y ago
        That's fairly reasonable. At least they can prompt you "Did you really mean to do this?" -- they have enough infrastructure to update your project page with a "Make pull request" button after you push, they can repurpose that for a "Approve next push without asking questions." Alternatively, if they supported custom pre-receive hooks via e.g. webhooks, someone could publish a script or stand up a web server to check for such things.
- scott_karana 11y ago
  > Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully For what it's worth, the word is "waived". :-) I'm glad Amazon dealt with you well!
- nadams 11y ago
  > I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. This paradigm is very puzzling to me. Why do people feel the need to publish every small project to the public? Is it because github is so easy to create a new repo? Why don't more people use private repos like self hosted or bitbucket? The whole reason why I spawned my own source code hosting service is so that I can work on projects in private without worrying about random people looking at what I'm working on (some of my C++ projects would give even Stroustrup a heart attack...). Note - this isn't saying bitbucket or gitlab or any of the others aren't good (oh I have my own opinions and comments about them...) - I've become rather bitter/paranoid/resentment of offers of free hosting.
  - akerl_ 11y ago
    I'm surprised that of the quoted comment you focus on the "paradigm" of open sourcing code, and not the "paradigm" of mixing credentials with code. Trying to protect against leaking creds by not open sourcing is a bit of misdirected effort.
    - nadams 11y ago
      I pointed this out because I've seen this over and over again with the single theme of "I was working on some silly project using AWS and I pushed my credentials to github". I don't think I've ever seen a professional team push AWS credentials (or any other credentials for that matter) - and if they did it's very rare to the point where I don't remember. As far as mixing credentials with code - that happens all the time. I'm not saying it's right but I have much bigger concerns - such as why people keep silly projects to github with their credentials. The first time I saw an article about it - I found out that there are bad people who are monitoring github for exactly that - and they will use it even before you realize it what you did. And not just AWS credentials mind you - MySQL, SSH, anything they can get their hands on... I think my comments are getting lost in translation - I'm not saying I'm against people pushing their silly projects. Push your 1 line GPL code projects all day long - I don't care. That's why github is there. What I find puzzling is not that people want to create new projects - but they feel like they must create a git repo for EVERYTHING they do and push it up to a public github like it's going to change the face of computer science as we know it or something. I've seen people keep documents and even bash profiles as public repos. I'm not saying it isn't a good idea to use git for those purposes - but I wouldn't want my bash profile common knowledge (especially if it contains commands or functions that I use at my day job).
- voltagex_ 11y ago
  I'm as close to a "casual" user of AWS as you can get. I find IAM incredibly difficult to use. Any pointers?
- joshuafalken_ 11y ago
  I created a small tool[1] to help continuously audit public github commits for secrets, like aws keys. It uses the AWS provided regexes[2] to do to. [1] - https://github.com/jfalken/github_commit_crawler https://github.com/jfalken/github_commit_crawler [2] - http://blogs.aws.amazon.com/security/blog/tag/key+rotation http://blogs.aws.amazon.com/security/blog/tag/key+rotation
- snorkel 11y ago
  The real mistake here is using your account owner API key in your code, when you shouldn't use that key for anything ... In fact you should delete the account owner access key!!! Otherwise you may have used the key for a Power User or Administrator level IAM user in your code instead of a custom user with least privileges. Always use least privileged IAM keys in code if you must, preferably use EC2 roles instead of access keys, and never give your IAM users permission to run instances ... That way if the key is leaked then evil hacker can't do much with it.
  - jebblue 11y ago
    I agree. I haven't used AWS for a while but IIRC for a small side project I was looking into with a couple of friends I set up a low privilege IAM access for them but I logged directly into the VPS using a password if I needed to update something.
cperciva 11y ago
For Tarsnap, I have several completely independent AWS accounts. This is partly because I needed privilege separation before IAM existed; but I keep this setup mainly because it's dead simple and completely avoids the risk of user error: When I'm doing development work, I don't even have access to the Tarsnap production accounts.
- gargarplex 11y ago
  Genius insight that could be applied to not just AWS but Parse, etc.
  - ksikka 11y ago
    Hey gargarplex - on the off chance that you see this, could you send me an email? Wanted to get in touch with you regarding one-to-many supplier registration. I've saved it in the about section of my user page. Apologies in advance for the unrelated comment.
- eropple 11y ago
  Yeah, this is the way to go. For when I have to use the AWS console, I use separate Chrome profiles for each of my environments, so I don't even have cookies, saved passwords, etc. that might relate to other accounts. (I also use separate Chrome themes in each environment and use a CSS manipulator such that my prod console is in a bright-freaking-red Chrome window with a red background. Blue for test, green for dev.) For my own AWS accounts, I use a read-only account in the profile and, on the very rare occasion I want to get clicky rather than use awscli to frob something, I'll log out and log back in as my writable-to-IAM user. I have a third user that only has IAM read/write privileges if I have to deal with that service. (I haven't sold my clients on this approach, though.)
  - icelancer 11y ago
    > use a CSS manipulator such that my prod console is in a bright-freaking-red Chrome window with a red background. Blue for test, green for dev Ha, I do the same thing. I learned about this from a previous job I had where prod/test webapps were functionally similar but had obnoxious coloring in the header. Very effective.
whisk3rs 11y ago
The MFA Condition is a huge win, and I'm surprised Amazon hasn't built this a tool to make this easier yet. However, I question the merit of using two separate AWS accounts. While this separation of responsibility sounds nice in theory, doesn't it introduce additional maintenance burden because you now have two accounts to administer? You can't define or manage the roles in the 2nd account without credentials to do so.
- Rapzid 11y ago
  Usually those other accounts have much fewer people with access=much lower risk of unauthorized access. For instance, I'm a big proponent of a backup acct that the main acct can push to but not delete from. That backup acct can have very limited and tightly controlled access. It's unfortunate RDS does you no favours in helping out with this; you pretty much have to dump your DB and push it off AWS or into another acct's S3 bucket. IAM is just a PITA is what this boils down to. Create an IAM policy that allows users to push updates to elastic beanstalk but not touch any other resources in the account.. It's a major, major hassle. AWS has no concept of resource groups and each service has different ways of restricting access(ec2 can do it on tags, other services you kinda have to use naming schemas and wild cards in your policies). So you are often left needing to have users with a little too much access, and/or spending a LOT of time testing and crafting IAM policies.. IAM is a really good idea and powerful in many ways but unfortunately AWS's lack of consistency and UX across individual services really shows through sometimes, and with IAM in particular.
- JamesLeonis 11y ago
  I recently switched to using the MFA service with Google's Authenticator app. I find it more pleasant than the normal send-text-to-device implementation. So far it comes with my recommendation.
Havoc 11y ago
//Somewhat off topic Using any kind of AWS account (in a personal capacity) is a serious risk in my eyes. Its much like I'm happy to take the risk of buying shares, but don't like leveraged derivatives. I don't want to deal with something that is not my area of expertise and has very real potential to blow up in my face. I wish Amazon allowed me to cap spending to say 200USD...that I can plan for much easier than X thousands.
- tokenizerrr 11y ago
  Agreed. Wish I could limit cap the monthly amount spent.
  - Havoc 11y ago
    Yup - frankly I'm very wary of anything that is an uncertain amount. Did the same with my cellphone contracts - take expected spend, triple it and make that the limit. Put a number on it (even triple) and I can deal with it...just not liable for "unknown".
- eropple 11y ago
  If you look around (or spend a few minutes writing one), you can use a scram switch that goes off of AWS billing alerts.
  - CJefferson 11y ago
    What if I'm asleep? Or it fails? Seems so much easier, and safer, for Amazon to just have a hard "don't charge more than this". Increasingly, I wish this could be fixed at the banking end -- I'd like to be able to get my bank to enforce that I won't give more than £X to some company (obviously the company should be able to find this fact out, and not give me products worth more than £X).
    - toomuchtodo 11y ago
      Use prepaid cards you load with a fixed sum.
astral303 11y ago
Another risk consideration is that anyone getting unauthorized access to your AWS account can delete all your resources and all your backups (snapshots, etc), effectively putting you out of business. [1] One solution is to backup to a separate backup-only AWS account, with super-serious access controls (MFA and password physically locked away somewhere). Set up a "write-only" link, such that backups can be added, but never removed. This way, in the worst case, your runtime infrastructure can be decimated, but your data backups would be safe. 1 - http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/ http://arstechnica.com/security/2014/06/aws-console-breach-l...
- badmadrad 11y ago
  I prefer this method. Having multipe aws dashboards sounds like a nightmare. I would rather use the backup account approach with MFA tied to the administrator accounts on each aws site.
  - eropple 11y ago
    Multiple AWS dashboards is actually really easy with multiple Chrome profiles (not that I go to them often, there are APIs for a reason) and tends to encourage much better, much more isolated application design. I've worked in environments with a single account and in environments with multiple accounts and I can't imagine going backwards.
- technion 11y ago
  I've recently setup Glacier with a Vault policy that prevents deletions. I really like that layer of protection, but I'm under no illusion as to what a disaster it would still be if the main pass was compromised.
deleted 11y ago
[deleted]
anu_gupta 11y ago
Is there an easier to use UI for setting up IAM users. I get hopelessly confused every time I attempt it, and worry about losing access to services I've already set up.
- fletchowns 11y ago
  I don't think so. Mine is always a painful experience of trial and error using a combination of things cobbled together from the policy generator, manual editing, copy/pasting from AWS documentation, and using the policy simulator. And then after hours of that I can determine that IAM policies fall way short off what I needed (isolation between different product groups in the org) and that we need to use separate root accounts.
kikibobo69 11y ago
At Zalando we have an account per team, and have been releasing a bunch of tooling to help do this securely.[1] It's not completely easy (e.g., account creation at Amazon can not be automated), but the security aspects are really nice, and it lets us give teams more or less full access to just their account. [1] https://stups.io https://stups.io
setheron 11y ago
What's cool is that in AWS as a developer they disable the ability to even login with username and password. You can set it up such that the only access is through a site that grants federated access.
developer1 11y ago
"If someone gets access to your AWS access credentials, you’re in trouble." I know we're not supposed to post negative comments that don't "add value" to a discussion, but the only thing that comes to mind is "really?". Your setup is as secure as you make it. How you use your API access is up to you. Putting all your eggs in one basket is not insecure. This article doesn't actually bring to light anything important. There is no risk involved so long as you pay attention to what you are doing. Any set of credentials, if leaked, destroys security. So... don't set yourself up to leak your credentials? I mean, come on, seriously? *edit: I unfairly used the word "incompetent". Change to a phrase about paying attention to what you are doing.
- zippergz 11y ago
  Pay attention to what you're doing, and never make a mistake. Or have anyone in your team/company make a mistake. While you're at it, please make sure you don't put any bugs in the code also. We're humans. We're imperfect. Sometimes a safety net isn't a bad thing.
  - angrybits 11y ago
    Except for the fact that you can explicitly contain the potential damage by controlling which accounts can do what. It's not like you have to have one god account with no MFA that has the keys to the city.
- snorkel 11y ago
  This is easy advice to a single DevOp who completely controls their own AWS account, but often there is a team of devs and ops with elevated access to the same AWS account and not all of them understand AWS IAM access control, so they put admin level keys in their code. This article gives advice for mitigating the risk of access key leakage.
hamza001 11y ago
http://googler700.blogspot.com/ http://googler700.blogspot.com/
alexchamberlain 11y ago
I seriously question any business running just on AWS. There are plenty of other services that "look like" AWS and give you complete isolation.
djhworld 11y ago
I think the comment about using temporary credentials is an interesting one. Where I work, we have a federated access system that generates temporary credentials so you can access the Console without having to sign in, and it uses our work authentication mechanism to do the initial handshake. The thing is this process is useless if you want to use it with the AWS command line client, or any other tools that rely on the use of credentials. It would be nice if AWS adopted a plugin system or something where you could plug in a "credentials provider" of some sort, and the command line client queries that for credentials each time you make a request, instead of having to stash keys in ~/.aws/credentials
evook 11y ago
aye aye captain obvious!
chris_wot 11y ago
Is there a good primer on AWS and what it provides?
autotune 11y ago
You can also use ssh forwarding to log into your servers through the SSH bastion without dropping your private keys everywhere on the bastion itself, securing access even further.
nodesocket 11y ago
I like the approach that Google cloud takes with projects. Projects are independent and isolated under a single account.

322 points by hellomichibye 11y ago

azinman2 11y ago

Like the solution via MFA & temp credentials! Simple & elegant.

deleted 11y ago

[deleted]

marblar 11y ago
> Why is there a tophat over "IAM role”? Presumably because rather than being a person, the IAM role is a “hat” that a person wears.

impostervt 11y ago

About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. The next day I got an email from Amazon, alerting me to the problem. Apparently, they scrape github looking for just that kind of stupidity. I instantly deleted the project, but it was too late. Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully. I'm now a zealous advocate for making sure a person never even HAS AWS credentials. Instead, make a new user without a password for each use case, and manually select the privileges that account has. If you have a password to AWS, you shouldn't have credentials.

e40 11y ago
I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things. What kills me is there is no easy way to stop all the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.
- pests 11y ago
  You can automate the stopping via the CLI interface. While I don't think there is a single command to stop all instances, you should be able to whip up a script to get all id's then call the stop-instances command on each of them.
- agumonkey 11y ago
  You must lose a fair amount of water when you're greeted with a surprise bill of $11k.
  - e40 11y ago
    Yeah, it's a rare moment in life. Especially fun since it happened on a Saturday morning (the notification, that is).
- copperx 11y ago
  When people find the accounts, what do they use them for? Mining bitcoin?
- mrestko 11y ago
  Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?
  - fpgaminer 11y ago
    It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...
    - icelancer 11y ago
      ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.
aikah 11y ago
> Coding fast and dumb, And that's why you should use Bibucket first and foremost. And then when you vet your stuff and you know it's clean, you can publish it on publicly Github. If you rush pushing code on github public repos that's exactly what will happen. If you start with a private one then move to a public one you have more time to think things through. Now one could imagine a third party service warning users of potential issues with files pushed,based on their name/extension/folder name. But privacy first , then open the code to the public.
- bosdev 11y ago
  You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. This solution is also strange because if you ever committed anything private it will be in the history. So to make this work you either have to rebase over some history, or lose all the history.
  - zippergz 11y ago
    I have a lot of private repos. One for just about every little project I've done. Some larger projects have more than one. It would be pretty expensive to keep all of this in Github that way. The alternative is to combine a bunch of unrelated stuff into fewer repos (or not keep all of my projects in source control), neither of which is especially appealing. I'd be happy to pay Github something for my usage and to support development. I believe in paying people for work that I find useful. But the pricing model makes the cost disproportionate with the value for me.
  - eropple 11y ago
    > You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. Github doesn't have a personal tier big enough for my private repos. (And I'd host my own before paying $50/month.) I do pay Bitbucket, though, because I have a consistent and small-enough group of collaborators that it makes sense.
  - belovedeagle 11y ago
0x0 11y ago
Github provides a public near-realtime stream of events, I'm sure blackhats are constantly monitoring it looking for private keys all the time. Even if you undo a commit within seconds, it may be too late. https://api.github.com/events https://api.github.com/events
- meddlepal 11y ago
  I did not know that existed and that is pretty awesome!
- unethical_ban 11y ago
  If Amazon (and my coworker in a 3 person dev shop) can automate the scanning of common API key patterns as a pre-commit filter, I wonder why Github itself doesn't flag for this sort of thing before publishing.
  - giaour 11y ago
    IIRC, a regex that matches a generic AWS access key will also match a git hash.
    - giaour 11y ago
      Not sure what I said that warrants multiple downvotes. Git commits are identified by SHA1 hashes, so they would be caught by the same regex that would catch AWS keys. According to the AWS security blog, this is the regex you should use for secret keys: `(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])`. To match git hashes, you can use this one: `[0-9a-f]{40}`. See the likeness?
  - IanCal 11y ago
    What should they do? Reject a "git push"?
    - geofft 11y ago
      That's fairly reasonable. At least they can prompt you "Did you really mean to do this?" -- they have enough infrastructure to update your project page with a "Make pull request" button after you push, they can repurpose that for a "Approve next push without asking questions." Alternatively, if they supported custom pre-receive hooks via e.g. webhooks, someone could publish a script or stand up a web server to check for such things.
scott_karana 11y ago
> Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully For what it's worth, the word is "waived". :-) I'm glad Amazon dealt with you well!
nadams 11y ago
> I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. This paradigm is very puzzling to me. Why do people feel the need to publish every small project to the public? Is it because github is so easy to create a new repo? Why don't more people use private repos like self hosted or bitbucket? The whole reason why I spawned my own source code hosting service is so that I can work on projects in private without worrying about random people looking at what I'm working on (some of my C++ projects would give even Stroustrup a heart attack...). Note - this isn't saying bitbucket or gitlab or any of the others aren't good (oh I have my own opinions and comments about them...) - I've become rather bitter/paranoid/resentment of offers of free hosting.
- akerl_ 11y ago
  I'm surprised that of the quoted comment you focus on the "paradigm" of open sourcing code, and not the "paradigm" of mixing credentials with code. Trying to protect against leaking creds by not open sourcing is a bit of misdirected effort.
  - nadams 11y ago
    I pointed this out because I've seen this over and over again with the single theme of "I was working on some silly project using AWS and I pushed my credentials to github". I don't think I've ever seen a professional team push AWS credentials (or any other credentials for that matter) - and if they did it's very rare to the point where I don't remember. As far as mixing credentials with code - that happens all the time. I'm not saying it's right but I have much bigger concerns - such as why people keep silly projects to github with their credentials. The first time I saw an article about it - I found out that there are bad people who are monitoring github for exactly that - and they will use it even before you realize it what you did. And not just AWS credentials mind you - MySQL, SSH, anything they can get their hands on... I think my comments are getting lost in translation - I'm not saying I'm against people pushing their silly projects. Push your 1 line GPL code projects all day long - I don't care. That's why github is there. What I find puzzling is not that people want to create new projects - but they feel like they must create a git repo for EVERYTHING they do and push it up to a public github like it's going to change the face of computer science as we know it or something. I've seen people keep documents and even bash profiles as public repos. I'm not saying it isn't a good idea to use git for those purposes - but I wouldn't want my bash profile common knowledge (especially if it contains commands or functions that I use at my day job).
voltagex_ 11y ago
I'm as close to a "casual" user of AWS as you can get. I find IAM incredibly difficult to use. Any pointers?
joshuafalken_ 11y ago
I created a small tool[1] to help continuously audit public github commits for secrets, like aws keys. It uses the AWS provided regexes[2] to do to. [1] - https://github.com/jfalken/github_commit_crawler https://github.com/jfalken/github_commit_crawler [2] - http://blogs.aws.amazon.com/security/blog/tag/key+rotation http://blogs.aws.amazon.com/security/blog/tag/key+rotation
snorkel 11y ago
The real mistake here is using your account owner API key in your code, when you shouldn't use that key for anything ... In fact you should delete the account owner access key!!! Otherwise you may have used the key for a Power User or Administrator level IAM user in your code instead of a custom user with least privileges. Always use least privileged IAM keys in code if you must, preferably use EC2 roles instead of access keys, and never give your IAM users permission to run instances ... That way if the key is leaked then evil hacker can't do much with it.
- jebblue 11y ago
  I agree. I haven't used AWS for a while but IIRC for a small side project I was looking into with a couple of friends I set up a low privilege IAM access for them but I logged directly into the VPS using a password if I needed to update something.

cperciva 11y ago

For Tarsnap, I have several completely independent AWS accounts. This is partly because I needed privilege separation before IAM existed; but I keep this setup mainly because it's dead simple and completely avoids the risk of user error: When I'm doing development work, I don't even have access to the Tarsnap production accounts.

gargarplex 11y ago
Genius insight that could be applied to not just AWS but Parse, etc.
- ksikka 11y ago
  Hey gargarplex - on the off chance that you see this, could you send me an email? Wanted to get in touch with you regarding one-to-many supplier registration. I've saved it in the about section of my user page. Apologies in advance for the unrelated comment.
eropple 11y ago
Yeah, this is the way to go. For when I have to use the AWS console, I use separate Chrome profiles for each of my environments, so I don't even have cookies, saved passwords, etc. that might relate to other accounts. (I also use separate Chrome themes in each environment and use a CSS manipulator such that my prod console is in a bright-freaking-red Chrome window with a red background. Blue for test, green for dev.) For my own AWS accounts, I use a read-only account in the profile and, on the very rare occasion I want to get clicky rather than use awscli to frob something, I'll log out and log back in as my writable-to-IAM user. I have a third user that only has IAM read/write privileges if I have to deal with that service. (I haven't sold my clients on this approach, though.)
- icelancer 11y ago
  > use a CSS manipulator such that my prod console is in a bright-freaking-red Chrome window with a red background. Blue for test, green for dev Ha, I do the same thing. I learned about this from a previous job I had where prod/test webapps were functionally similar but had obnoxious coloring in the header. Very effective.

whisk3rs 11y ago

The MFA Condition is a huge win, and I'm surprised Amazon hasn't built this a tool to make this easier yet. However, I question the merit of using two separate AWS accounts. While this separation of responsibility sounds nice in theory, doesn't it introduce additional maintenance burden because you now have two accounts to administer? You can't define or manage the roles in the 2nd account without credentials to do so.

Rapzid 11y ago
Usually those other accounts have much fewer people with access=much lower risk of unauthorized access. For instance, I'm a big proponent of a backup acct that the main acct can push to but not delete from. That backup acct can have very limited and tightly controlled access. It's unfortunate RDS does you no favours in helping out with this; you pretty much have to dump your DB and push it off AWS or into another acct's S3 bucket. IAM is just a PITA is what this boils down to. Create an IAM policy that allows users to push updates to elastic beanstalk but not touch any other resources in the account.. It's a major, major hassle. AWS has no concept of resource groups and each service has different ways of restricting access(ec2 can do it on tags, other services you kinda have to use naming schemas and wild cards in your policies). So you are often left needing to have users with a little too much access, and/or spending a LOT of time testing and crafting IAM policies.. IAM is a really good idea and powerful in many ways but unfortunately AWS's lack of consistency and UX across individual services really shows through sometimes, and with IAM in particular.
JamesLeonis 11y ago
I recently switched to using the MFA service with Google's Authenticator app. I find it more pleasant than the normal send-text-to-device implementation. So far it comes with my recommendation.

Havoc 11y ago

//Somewhat off topic Using any kind of AWS account (in a personal capacity) is a serious risk in my eyes. Its much like I'm happy to take the risk of buying shares, but don't like leveraged derivatives. I don't want to deal with something that is not my area of expertise and has very real potential to blow up in my face. I wish Amazon allowed me to cap spending to say 200USD...that I can plan for much easier than X thousands.

tokenizerrr 11y ago
Agreed. Wish I could limit cap the monthly amount spent.
- Havoc 11y ago
  Yup - frankly I'm very wary of anything that is an uncertain amount. Did the same with my cellphone contracts - take expected spend, triple it and make that the limit. Put a number on it (even triple) and I can deal with it...just not liable for "unknown".
eropple 11y ago
If you look around (or spend a few minutes writing one), you can use a scram switch that goes off of AWS billing alerts.
- CJefferson 11y ago
  What if I'm asleep? Or it fails? Seems so much easier, and safer, for Amazon to just have a hard "don't charge more than this". Increasingly, I wish this could be fixed at the banking end -- I'd like to be able to get my bank to enforce that I won't give more than £X to some company (obviously the company should be able to find this fact out, and not give me products worth more than £X).
  - toomuchtodo 11y ago
    Use prepaid cards you load with a fixed sum.

astral303 11y ago

Another risk consideration is that anyone getting unauthorized access to your AWS account can delete all your resources and all your backups (snapshots, etc), effectively putting you out of business. [1] One solution is to backup to a separate backup-only AWS account, with super-serious access controls (MFA and password physically locked away somewhere). Set up a "write-only" link, such that backups can be added, but never removed. This way, in the worst case, your runtime infrastructure can be decimated, but your data backups would be safe. 1 - http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/ http://arstechnica.com/security/2014/06/aws-console-breach-l...

badmadrad 11y ago
I prefer this method. Having multipe aws dashboards sounds like a nightmare. I would rather use the backup account approach with MFA tied to the administrator accounts on each aws site.
- eropple 11y ago
  Multiple AWS dashboards is actually really easy with multiple Chrome profiles (not that I go to them often, there are APIs for a reason) and tends to encourage much better, much more isolated application design. I've worked in environments with a single account and in environments with multiple accounts and I can't imagine going backwards.
technion 11y ago
I've recently setup Glacier with a Vault policy that prevents deletions. I really like that layer of protection, but I'm under no illusion as to what a disaster it would still be if the main pass was compromised.

deleted 11y ago

[deleted]

anu_gupta 11y ago

Is there an easier to use UI for setting up IAM users. I get hopelessly confused every time I attempt it, and worry about losing access to services I've already set up.

fletchowns 11y ago
I don't think so. Mine is always a painful experience of trial and error using a combination of things cobbled together from the policy generator, manual editing, copy/pasting from AWS documentation, and using the policy simulator. And then after hours of that I can determine that IAM policies fall way short off what I needed (isolation between different product groups in the org) and that we need to use separate root accounts.

kikibobo69 11y ago

At Zalando we have an account per team, and have been releasing a bunch of tooling to help do this securely.[1] It's not completely easy (e.g., account creation at Amazon can not be automated), but the security aspects are really nice, and it lets us give teams more or less full access to just their account. [1] https://stups.io https://stups.io

setheron 11y ago

What's cool is that in AWS as a developer they disable the ability to even login with username and password. You can set it up such that the only access is through a site that grants federated access.

developer1 11y ago

"If someone gets access to your AWS access credentials, you’re in trouble." I know we're not supposed to post negative comments that don't "add value" to a discussion, but the only thing that comes to mind is "really?". Your setup is as secure as you make it. How you use your API access is up to you. Putting all your eggs in one basket is not insecure. This article doesn't actually bring to light anything important. There is no risk involved so long as you pay attention to what you are doing. Any set of credentials, if leaked, destroys security. So... don't set yourself up to leak your credentials? I mean, come on, seriously? *edit: I unfairly used the word "incompetent". Change to a phrase about paying attention to what you are doing.

zippergz 11y ago
Pay attention to what you're doing, and never make a mistake. Or have anyone in your team/company make a mistake. While you're at it, please make sure you don't put any bugs in the code also. We're humans. We're imperfect. Sometimes a safety net isn't a bad thing.
- angrybits 11y ago
  Except for the fact that you can explicitly contain the potential damage by controlling which accounts can do what. It's not like you have to have one god account with no MFA that has the keys to the city.
snorkel 11y ago
This is easy advice to a single DevOp who completely controls their own AWS account, but often there is a team of devs and ops with elevated access to the same AWS account and not all of them understand AWS IAM access control, so they put admin level keys in their code. This article gives advice for mitigating the risk of access key leakage.

hamza001 11y ago

http://googler700.blogspot.com/ http://googler700.blogspot.com/

alexchamberlain 11y ago

I seriously question any business running just on AWS. There are plenty of other services that "look like" AWS and give you complete isolation.

djhworld 11y ago

I think the comment about using temporary credentials is an interesting one. Where I work, we have a federated access system that generates temporary credentials so you can access the Console without having to sign in, and it uses our work authentication mechanism to do the initial handshake. The thing is this process is useless if you want to use it with the AWS command line client, or any other tools that rely on the use of credentials. It would be nice if AWS adopted a plugin system or something where you could plug in a "credentials provider" of some sort, and the command line client queries that for credentials each time you make a request, instead of having to stash keys in ~/.aws/credentials

evook 11y ago

aye aye captain obvious!

chris_wot 11y ago

Is there a good primer on AWS and what it provides?

autotune 11y ago

You can also use ssh forwarding to log into your servers through the SSH bastion without dropping your private keys everywhere on the bastion itself, securing access even further.

nodesocket 11y ago

I like the approach that Google cloud takes with projects. Projects are independent and isolated under a single account.