10 ms·

EC2's most dangerous feature

256 points by dwaxe 10y ago

cperciva 10y ago
OK, dwaxe, I have to ask: Are you a robot? Because I uploaded this blog post, tweeted it, and then came straight over here to submit it and you still got here first. Not that I mind, but getting your HN submission in within 30 seconds of my blog post going up is very impressive if you're not a robot.
- brettproctor 10y ago
  Assuming dwaxe replies claiming to not be a robot, how would we go about verifying? :P
  - ATsch 10y ago
    I heard robots float on water
    - bsandert 10y ago
      What also floats on water?
      - tomludus 10y ago
        Ducks!
      - OJFord 10y ago
        Did I mention that dwaxe turned me into a newt? (I got better.)
  - pilsetnieks 10y ago
    We could administer it a test of some sort, and evaluate the ensuing conversation, whether it's convincingly human. I think it'd be fitting name this test in honor of some forefather of computer science.
  - lucideer 10y ago
    We could compare the time it took to submit (seconds) to the time it took to comment (3 hours and counting...)
- human_error 10y ago
  Probably right time right place.
- jacquesm 10y ago
  Suggestion: post before tweeting. And to your question, dwaxe is not a bot (there are comments associated with the account too), and this has happened before (apparently a lightning fast submitter): https://news.ycombinator.com/item?id=12218797 https://news.ycombinator.com/item?id=12218797 Of course he/she could still be running a script.
  - cperciva 10y ago
    Suggestion: post before tweeting. I'll probably do that next time. I didn't think it would matter! (And really it doesn't -- it's not as if I need the karma points.)
    - chrisan 10y ago
      > And really it doesn't -- it's not as if I need the karma points. It's the principle of the matter! Does HN allow bot submissions?
  - OJFord 10y ago
    Or better; assuming the URI is known before posting (it's not randomly generated on submission or encapsulates absurdly precise submission time) just post it to HN a fraction of a second before it's live. Nothing any script can do to beat you then.
    - jacquesm 10y ago
      HN doesn't check submissions by fetching them so that would work.
- Scaevolus 10y ago
  Yes, there's definitely a script at work. Compare these timestamps (hover over date): https://hn.algolia.com/?query=dwaxe%20eff&sort=byPopularity&prefix&page=0&dateRange=pastWeek&type=story https://hn.algolia.com/?query=dwaxe%20eff&sort=byPopularity&... To the RSS feed: https://www.eff.org/rss/updates.xml https://www.eff.org/rss/updates.xml Humans don't post everything from an RSS feed within 10 minutes!
  - 0x0 10y ago
    Wouldn't it be funny if you added an RSS entry that linked to an otherwise unpublished page with the url path /i-am-a-bot-that-posts-to-hackernews with a matching title to go?
    - cperciva 10y ago
      Looks like that worked: https://news.ycombinator.com/item?id=12673881 https://news.ycombinator.com/item?id=12673881
      - 0x0 10y ago
        Haha that's brilliant! Nice touch with the recursive link.
  - jacques_chester 10y ago
    > Yes, there's definitely a script at work. Presumably to farm karma for some other purpose. I once observed that everything that shows up here eventually shows up in a particular cluster of subreddits and vice versa. Usually with a lag of 24-48 hours. I half-jokingly floated the idea that one could write a karma arbitrageur bot which cross posts between HN and those subreddits. I was told that I would be banned. Not just the bot: me.
- jakozaur 10y ago
  Most likely dwaxe is human using scripts to boost his karma. Almost like sportsman using enhancing performance drugs. Side question: I wonder when AI pass Hacker News Turing test. So bot can trick us into being human by HN comments.
- 0xmohit 10y ago
  The submissions [0] appear to be scripted. Most of those appear to be from selected few domains. Your domain seems to have been added recently [1]. Congratulations! So whether you choose to blog about tarsnap or anything else, chances are that it'd be posted to HN before you're able to. [0] https://news.ycombinator.com/submitted?id=dwaxe https://news.ycombinator.com/submitted?id=dwaxe [1] https://news.ycombinator.com/from?site=daemonology.net https://news.ycombinator.com/from?site=daemonology.net
- latentpot 10y ago
  Could be using something like IFTTT with a simple rule RSS to Post Immediately on Forum
- jacques_chester 10y ago
  > OK, dwaxe, I have to ask: Are you a robot? Cyborg. Story submission is clearly robotic, but there's a lot of charmingly humanesque entries under dwaxe's comments: https://news.ycombinator.com/threads?id=dwaxe https://news.ycombinator.com/threads?id=dwaxe
- dwaxe 10y ago
  Yes I am. This is my personal account, but I use it to automatically post to Hacker News. I was playing around with BigQuery one day and found the Hacker News dataset [1]. From my experience with the Reddit submissions dataset [2], I knew that I could compose this query, SELECT AVG(score) AS avg_score, COUNT(* ) AS num, REGEXP_EXTRACT(url, r'//([^/]*)/') AS domain FROM [fh-bigquery:hackernews.full_201510] WHERE score IS NOT NULL AND url <> '' GROUP BY domain HAVING num > 10 ORDER BY avg_score DESC which returns a list of domains with more than ten submissions sorted by average score. This turns out to be a list of some of the most successful tech blogs on the internet, as well as various YCombinator related materials. Out of the domains with over 100 submissions, daemonology.net has the 9th highest average score per submission. I manually visited all the domains with more than about 30 submissions, found the appropriate xml feeds, and saved them. I added a few websites like eff.org whose messages I think everyone should read anyways. Then I jumped into python and started trying to figure out how to post to Hacker News. It was a little more complicated than I anticipated [3], but an open source HN app for Android helped me figure it out. I set up a cron job on my $5 Digital Ocean that runs the script every few minutes (pseudocode): If you can reach http://news.ycombinator.com http://news.ycombinator.com, Check all feeds for new entries, Post a new entry to hn, Sleep for an hour before posting another [1] https://bigquery.cloud.google.com/dataset/bigquery-public-data:hacker_news https://bigquery.cloud.google.com/dataset/bigquery-public-da... [2] The only difference on Reddit is the subreddit system. [3] After you send a POST request to send to the login screen, Hacker News gives you a url with a unique "fnid" parameter, and you send another POST request to another url with the appropriate "fnid".
  - dang 10y ago
    We appreciate both the cleverness here and your detailed explanation. But could you please not do this anymore? It isn't malicious, but it's unhealthy for HN's ecosystem. For example, when an author submits his or her own work, that can add a lot of value to the community—but your bot pre-empts that, as it indeed did in this case. There are many more reasons why this isn't a good thing for HN. For example, it's better for submissions from popular sites to be distributed across a wide range of accounts. That gives more users a chance to feel like they're making important contributions, and gives the community (and authors) a clearer sense of the audience. There are lots of ways to write software to interact with HN, and lots of users with the ability to do it, so we really depend on the good will of the community only to do that when it serves the whole.
deleted 10y ago
[deleted]
Rapzid 10y ago
I've used firewall rules in the past to scope the metadata store to admin users.
- cperciva 10y ago
  That's better than nothing, but runs into problems if you have different users who need to be able to access different subsets of the metadata store.
  - cesarb 10y ago
    An admin user could fetch these subsets of the metadata and leave a copy of them in the local filesystem.
    - cperciva 10y ago
      That doesn't work for IAM Roles, because (a) AWS library code expects to get the keys out of the metadata store, and (b) IAM Role credentials are periodically rotated, so the keys you downloaded in advance would expire.
      - Rapzid 10y ago
        You're right it doesn't. IAM roles are intended to be granted to the entire server, not a subset of the server's users. Any compromise of the server would be considered a compromise of its role. Yeah, this is a bit crazy depending on what you're running on it. I was running multi-tenant IIS hosts and the apps had no business with the metadata or ec2 IAM roles in my case. If you want roles to work for other users via the meta data store you can intercept requests with a proxy and then grab the temp credentials STS assume role. This is how kube2iam works. Depending on your use case you'd have to write the proxy, automate the mappings and firewall rules, etc etc etc. PITA but probably doable. On a different note, I agree with just about everything you had to say about the PITA that is IAM. Properly scoping permissions is much harder than it needs to be. Not all resources support tags, and even then almost nothing outside of ec2 supports tag conditions in IAM. This leads to many naming schemas and wildcard resource conditions :( AWS should have created the concept of resource groups. This would have greatly simplified giving users permissions to subsets of an accounts resources. I chalk this up to AWS's VERY poor collaboration between service teams. Nothing that came out seemed coordinated. This appears to be getting better (shrug).
0x0 10y ago
This is interesting! Can this be abused with AWS-hosted services that reach out to fetch URLs? For example, image hosts that allow specifying an URL to retrieve, or OAuth callbacks, etc? Are there any tricks to be played if someone were to register a random domain and point it to 169.254.169.254 (or worse, flux between 169.254.169.254 and a public IP in case there is blacklisting application code that first checks to resolve the hostname but then passes the whole URL into a library that resolves again?)
- cperciva 10y ago
  The EC2 documentation specifically warns against allowing yourself to be tricked into loading your metadata for someone else. So yes, there are probably lots of services which get this wrong; but they did at least get a warning about that particular failure mode.
- idunno246 10y ago
  Yes, one project I've worked on required a crawling bot, and you could crawl the metadata service...(until fixed). dont even need a domain in most cases, either the ip or instance-data DNS I bet works in a bunch of places. We redid all our policies to be extremely restrictive in response, if the instance did anything based on user input. Anything more admin happened on a different machine.
- pfg 10y ago
  That's a fairly common vulnerability. A good approach for services that need to fetch arbitrary URLs is roughly: 1. Resolve hostname and remember the response 2. Verify that the response does not contain any addresses in a private IP space, or any other IP that is only accessible to you 3. Use the IP from step 1 when establishing a connection With other solutions, you might end up being vulnerable to DNS rebinding attacks. Bonus points for doing all your URL fetching in some sort of sandbox that enforces these access rules.
  - jamiesonbecker 10y ago
    This is accurate. Remember that even ELB's in AWS have IP's that change all the time, and this itself is actually a source of vulnerabilities from apps that don't respect DNS TTL's (as has been seen in the forums repeatedly -- apps get connected to the previous IP instead of the new one). It's probably safer to retrieve and verify the IP for each request, and just cache if the IP is 'safe'. (And just doing IP subnet calculations is non-trivial in most less-common languages.) Also, request throttling should be maintained and HTTP verb checking, to prevent being turned into a proxy for other attacks. Actually, any decision to accept an arbitrary URL should be carefully examined in light of how hard it is to do safely.
tex0 10y ago
It's much like the same Problem with the Google Cloud. Even worse there I'd say.
- sgrytoyr 10y ago
  Could you please elaborate? I'm not doubting you, just very interested in learning more.
- boulos 10y ago
  I just double checked, and the most similar thing we expose is the token's for each service-account in the instance metadata. As pointed out in the article, any uid on the box can read that. But, you can create instances with a zero-permission service account (the equivalent of nobody?) and just avoid it. This does mean that everywhere else you'd have to have explicit service accounts and such, but that seems like a reasonable "workaround" until or unless we make metadata access more granular (I like the block device idea! Would you want entirely different paths for JSON versus "plain" though?)
  - lobster_johnson 10y ago
    Google Cloud does seem better here. The exception is GKE — Kubernetes nodes are associated with service accounts which have permissions that, if abused by a malicious Docker container, could be disasterous for your entire cluster. Considering the amount of unpatched Docker containers out there, that's a bit scary. It also effectively prevents GKE from being usable in any scenario where you want to schedule containers on behalf of third-party actors (think PaaS). (GKE also doesn't let you disable privileged Docker containers, but that's another story.) On AWS you can run a metadata proxy to prevent pods from getting the credentials, but I don't know of a clean way to accomplish the same thing on GKE.
yandie 10y ago
If you're sharing the same instance for multiple users, trying to achieve security among the users is almost impossible anyway. That's why physical separation/virtualization is one of the first thing to focus on when talking about security.
- jamiesonbecker 10y ago
  Isolation is definitely important, but not all parts of the system running a single function need the same levels of access, and in fact it may be possible to target those components separately. Take a look at the wikipedia articles for 'defense in depth' or 'privilege separation' to see how important it is inside a system to treat each component isolated to itself as much as possible. (This is also why you don't want to rely on only a perimeter firewall for access control.)
gtsteve 10y ago
This is an interesting attack that I must confess I hadn't thought of, but surely any service that accepts an arbitrary URL has a list of IP ranges to avoid. However, to harden a role in the event of instance role credentials leaking, you could use an IAM Condition [0]. There is actually an example of this in the IAM documentation [1], although the source VPC parameter doesn't work for all services, and I can't see a list of services that support this parameter. This would ensure that the requests actually came from instances within your VPC. [0] http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html#Condition http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_po... [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies-vpc-endpoint.html#example-bucket-policies-restrict-access-vpc http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucke...
- jamiesonbecker 10y ago
  The point is not requests that originate elsewhere. The point is that this system is not protected in any way from any other process on your system.
- BraveNewCurency 10y ago
  Er, the problem is not "people can hit an un-routable IP from outside your instance". The problem is that "if your instance allows an attacker to make a HTTP request, you might expose personal information". For example, web crawlers or other fetchers.
logronoide 10y ago
Same happens with metadata access in Openstack. The access is controlled by source IP (and namespace). I wonder if it's possible to spoof the IP and access Metadata of other servers/users.
djb_hackernews 10y ago
If users can issue arbitrary commands on an instance then that instance should have zero Iam roles and should delegate actions to services running on separate instances. The instances hosting our users go a step further and null route Metadata service requests via iptables.
- jeremyjh 10y ago
  It isn't just about users, its also about malicious software you may accidentally install, if for example a library you use is compromised as has happened before with Ruby gems.
cesarb 10y ago
What I've done for a previous company was to, as one of the very first things done within every EC2 instance, add an iptables owner match rule to only allow packets destined to 169.254.169.254 if they come from uid 0. Any information from that webservice that non-root users might need (for instance, the EC2 instance ID) is fetched on boot by a script running as root and left somewhere in the filesystem.
- falcolas 10y ago
  This won't help with IAM roles, since the credentials provided in the metadata expire. Of course, a small tweak to the iptables entry would help there as well. Mind posting your entry for us iptables impared folks?
  - cesarb 10y ago
    I don't work there anymore, so I don't have access to the exact rule I used, but IIRC it was something like iptables -t filter -I OUTPUT -d 169.254.169.254 -m owner \! --uid-owner 0 -j REJECT --reject-with icmp-admin-prohibited
- jeremyjh 10y ago
  That is the obvious answer, do you have any scripts you could share?
- manojlds 10y ago
  But the data is not static.
  - abofh 10y ago
    Most of it is - your instance ID, network position, etc, will never change after boot (or 'start' if you stop it) - so caching it is just fine. There's very little that will change on a running instance except of course, the IAM credentials referred to in this article (as they expire within 90 minutes IIRC).
    - 10y ago
patsplat 10y ago
EC2 instances are designed to enforce isolation between instances, not processes. Presumably there would only be one primary service running on each. Use AWS be pushed towards an architecture based on containers and services. AWS is the OS, not any individual machine.
hueving 10y ago
The blog post buries the lead a little bit because it's talking about lots of pain points with the ec2 API and IAM. The important point to take away is that any process with network access running on your instance can contact the EC2 metadata service at http://169.254.169.254 http://169.254.169.254 and get the instance-specific IAM credentials. Think about things like services that accept user submitted URLs, crawl them, and display results...
- kevsim 10y ago
  I believe Pocket faced exactly this issue once upon a time.
  - Qerub 10y ago
    Indeed! https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-pocket/ https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-po...
- strictfp 10y ago
  ....and that you can imitate the metadata service to make life easier :) A plug for my friends side project: https://github.com/otm/limes https://github.com/otm/limes . It's a local metadataservice. Very handy for making aws libs work without having to config them much. And great support for MFAs.
- deleted 10y ago
  [deleted]
- novaleaf 10y ago
  do you know if any similar vulnerability exists with Azure or Google Cloud? edit: not sure why the downvote, I use Google Cloud, so honest question :(
  - Pharaoh2 10y ago
    You are getting downvoted because it's not a vulnerability. It's well documented and very useful feature in aws and it also exists in gcloud, although gcloud documentation is not as good^1. It only becomes a vulnerability if you don't read the documentation AND don't follow best practices^2. I don't know if azure provides this feature. ^1 I use gcloud now and have used aws in the past. ^2 Don't run unknown code without proper sandboxing
rcaught 10y ago
> almost as trivial for EC2 instances to expose XenStore as a filesystem to which standard UNIX permissions could be applied, providing IAM Role credentials with the full range of access control functionality which UNIX affords to files stored on disk. Doesn't this become more complicated when you think about EC2 offering Windows instances? Even with straight UNIX file writing, what writes this? Where does it write this? Which user has read permissions?
- skywhopper 10y ago
  Yeah, having the metadata available over an http interface is actually brilliant. Simple HTTP calls are easy to do from any network-capable OS or language.
  - jamiesonbecker 10y ago
    So is reading a file on the filesystem.. easier, actually in most languages, since HTTP requests usually require loading an extra library.
    - ChoHag 10y ago
      A library which uses the same (built-in) underlying IO mechanisms as the (built-in) filesystem.
- jamiesonbecker 10y ago
  In UNIX, the same way that EBS volumes are mounted... think of the /proc or /sys virtual filesystems. In Windows, I'm guessing that this would be exposed as a network drive.
  - rcaught 10y ago
    My point is that these type of solutions create a lot more overhead, inconsistency and variation compared to a HTTP request; granted, less security.
skywhopper 10y ago
Hopefully the operators using EC2 instance profiles understand and weigh the risks of using that feature. It's good to be cautious, but the feature is only dangerous if you don't take the time to understand it. Running a server on the Internet at all is "dangerous" in the same sense. And for this particular risk, it turns out there's a simple fix. He _is_ right in his first criticism that the IAM access controls available for much of the AWS API are entirely inadequate. In the case of EC2 in particular, it's all or nothing--either your credentials can call the TerminateInstances API or they can't. I'm sure Amazon is working on improving things, but for now it's pretty terrible. But in practice it just means you have to take care in different ways than you would if his tag-based authz solution were implemented. That said, while it's certainly frustrating to an implementor, it's not "dangerous" that limitations exist in these APIs. We're talking about decade-old APIs from the earliest days of AWS, and while things have been added, the core APIs are still the same. That's an amazing success story. But like any piece of software, there are issues that experienced users learn how to work around. You can bet that the EC2 API code is hard and scary to deal with for its maintainers. Adding a huge new permissions scheme is likely nearly impossible without a total rewrite... I don't envy them their task.
- jamiesonbecker 10y ago
  It's impossible to limit access to any part of the instance metadata in any way w/o firewalling (which has its own issues) or even to expire access to any part of it. Since instance profiles have keys (even though automatically rotated), any process on the system, owned by any user, can access anything exposed via the instance role. This makes embedding IAM keys into your instance and protecting it by root-only or ACL's MUCH MUCH safer... but AWS specifically states that instance profiles are preferred. In fact, for our Userify AWS instances (ssh key management), we are required to use instance roles and not allowed to offer the option. (This is why we do not offer S3 bucket storage on our AWS instances but we do on Pro and Enterprise self-hosted.) The biggest issue with the IAM instance profiles is that they trade security for convenience.. and it's not a good trade.
  - subway 10y ago
    For the most part EC2 instances should be single-purpose. Use tiny instances that do one job. Your IAM role describes the permissions that should be granted to that one job. It's absolutely true that you cannot isolate permissions at the process level, but by using single-job-type instances, you can easily isolate permissions on a per-job (in this model, per-instance) basis.
_hyn3 10y ago
tl;dr: 1) IAM instance roles have no security mechanisms to protect them from being read from any process on the instance, thus completely eliminating them from all Linux/UNIX/Windows permission systems. (The real reason for this is that instance metadata was a convenient semi-public information store for things like instance ID, but it was extended to also provide secret material, which was, at best, an idiotic move.) As the author points out, Xen already provided a great filesystem alternative that could be mounted as another drive (or network drive) to be managed with the regular OS filesystem permission system. (reading an instance ID is just a matter of reading a "file")... for some reason, AWS didn't leverage this and instead just added the secret material to its local instance metadata webserver. 2) the API calls are not fine grained enough and/or there are big holes in their coverage -- so, for instance, if you want to use some other AWS services, you can end up exposing much more than you intended.
deleted 10y ago
[deleted]
jamiesonbecker 10y ago
Until AWS fixes this (which, as the article points out, may never happen), a chmod'ed 600 file (only readable by root) is actually much safer, even when STS auto-rotation is taken into account.
icedchai 10y ago
IAM instance roles are still an improvement over how it was typically done in the past: hard-coding the same key in a configuration file and deploying it everywhere. It's a balance between security and convenience.
ChoHag 10y ago
The number of people in this thread not merely nodding their heads and mmhmm-ing (or the internet equivalent) is a concern.
zimbatm 10y ago
I wonder how many services on Amazon allow user-configurable webhooks that can be pointed to http://169.254.169.254 http://169.254.169.254 ...
Thaxll 10y ago
It has been the case for 10 years, anyone knows that, I don't see the problem. If you're not happy with it just use API keys.
- helper 10y ago
  IAM instance roles were only added in 2012: https://aws.amazon.com/blogs/aws/iam-roles-for-ec2-instances-simplified-secure-access-to-aws-service-apis-from-ec2/ https://aws.amazon.com/blogs/aws/iam-roles-for-ec2-instances...

256 points by dwaxe 10y ago

OK, dwaxe, I have to ask: Are you a robot? Because I uploaded this blog post, tweeted it, and then came straight over here to submit it and you still got here first. Not that I mind, but getting your HN submission in within 30 seconds of my blog post going up is very impressive if you're not a robot.

brettproctor 10y ago
Assuming dwaxe replies claiming to not be a robot, how would we go about verifying? :P
- ATsch 10y ago
  I heard robots float on water
  - bsandert 10y ago
    What also floats on water?
    - tomludus 10y ago
      Ducks!
    - OJFord 10y ago
      Did I mention that dwaxe turned me into a newt? (I got better.)
- pilsetnieks 10y ago
  We could administer it a test of some sort, and evaluate the ensuing conversation, whether it's convincingly human. I think it'd be fitting name this test in honor of some forefather of computer science.
- lucideer 10y ago
  We could compare the time it took to submit (seconds) to the time it took to comment (3 hours and counting...)
human_error 10y ago
Probably right time right place.
jacquesm 10y ago
Suggestion: post before tweeting. And to your question, dwaxe is not a bot (there are comments associated with the account too), and this has happened before (apparently a lightning fast submitter): https://news.ycombinator.com/item?id=12218797 https://news.ycombinator.com/item?id=12218797 Of course he/she could still be running a script.
- cperciva 10y ago
  Suggestion: post before tweeting. I'll probably do that next time. I didn't think it would matter! (And really it doesn't -- it's not as if I need the karma points.)
  - chrisan 10y ago
    > And really it doesn't -- it's not as if I need the karma points. It's the principle of the matter! Does HN allow bot submissions?
- OJFord 10y ago
  Or better; assuming the URI is known before posting (it's not randomly generated on submission or encapsulates absurdly precise submission time) just post it to HN a fraction of a second before it's live. Nothing any script can do to beat you then.
  - jacquesm 10y ago
    HN doesn't check submissions by fetching them so that would work.
Scaevolus 10y ago
Yes, there's definitely a script at work. Compare these timestamps (hover over date): https://hn.algolia.com/?query=dwaxe%20eff&sort=byPopularity&prefix&page=0&dateRange=pastWeek&type=story https://hn.algolia.com/?query=dwaxe%20eff&sort=byPopularity&... To the RSS feed: https://www.eff.org/rss/updates.xml https://www.eff.org/rss/updates.xml Humans don't post everything from an RSS feed within 10 minutes!
- 0x0 10y ago
  Wouldn't it be funny if you added an RSS entry that linked to an otherwise unpublished page with the url path /i-am-a-bot-that-posts-to-hackernews with a matching title to go?
  - cperciva 10y ago
    Looks like that worked: https://news.ycombinator.com/item?id=12673881 https://news.ycombinator.com/item?id=12673881
    - 0x0 10y ago
      Haha that's brilliant! Nice touch with the recursive link.
- jacques_chester 10y ago
  > Yes, there's definitely a script at work. Presumably to farm karma for some other purpose. I once observed that everything that shows up here eventually shows up in a particular cluster of subreddits and vice versa. Usually with a lag of 24-48 hours. I half-jokingly floated the idea that one could write a karma arbitrageur bot which cross posts between HN and those subreddits. I was told that I would be banned. Not just the bot: me.
jakozaur 10y ago
Most likely dwaxe is human using scripts to boost his karma. Almost like sportsman using enhancing performance drugs. Side question: I wonder when AI pass Hacker News Turing test. So bot can trick us into being human by HN comments.
0xmohit 10y ago
The submissions [0] appear to be scripted. Most of those appear to be from selected few domains. Your domain seems to have been added recently [1]. Congratulations! So whether you choose to blog about tarsnap or anything else, chances are that it'd be posted to HN before you're able to. [0] https://news.ycombinator.com/submitted?id=dwaxe https://news.ycombinator.com/submitted?id=dwaxe [1] https://news.ycombinator.com/from?site=daemonology.net https://news.ycombinator.com/from?site=daemonology.net
latentpot 10y ago
Could be using something like IFTTT with a simple rule RSS to Post Immediately on Forum
jacques_chester 10y ago
> OK, dwaxe, I have to ask: Are you a robot? Cyborg. Story submission is clearly robotic, but there's a lot of charmingly humanesque entries under dwaxe's comments: https://news.ycombinator.com/threads?id=dwaxe https://news.ycombinator.com/threads?id=dwaxe
dwaxe 10y ago
Yes I am. This is my personal account, but I use it to automatically post to Hacker News. I was playing around with BigQuery one day and found the Hacker News dataset [1]. From my experience with the Reddit submissions dataset [2], I knew that I could compose this query, SELECT AVG(score) AS avg_score, COUNT(* ) AS num, REGEXP_EXTRACT(url, r'//([^/]*)/') AS domain FROM [fh-bigquery:hackernews.full_201510] WHERE score IS NOT NULL AND url <> '' GROUP BY domain HAVING num > 10 ORDER BY avg_score DESC which returns a list of domains with more than ten submissions sorted by average score. This turns out to be a list of some of the most successful tech blogs on the internet, as well as various YCombinator related materials. Out of the domains with over 100 submissions, daemonology.net has the 9th highest average score per submission. I manually visited all the domains with more than about 30 submissions, found the appropriate xml feeds, and saved them. I added a few websites like eff.org whose messages I think everyone should read anyways. Then I jumped into python and started trying to figure out how to post to Hacker News. It was a little more complicated than I anticipated [3], but an open source HN app for Android helped me figure it out. I set up a cron job on my $5 Digital Ocean that runs the script every few minutes (pseudocode): If you can reach http://news.ycombinator.com http://news.ycombinator.com, Check all feeds for new entries, Post a new entry to hn, Sleep for an hour before posting another [1] https://bigquery.cloud.google.com/dataset/bigquery-public-data:hacker_news https://bigquery.cloud.google.com/dataset/bigquery-public-da... [2] The only difference on Reddit is the subreddit system. [3] After you send a POST request to send to the login screen, Hacker News gives you a url with a unique "fnid" parameter, and you send another POST request to another url with the appropriate "fnid".
- dang 10y ago
  We appreciate both the cleverness here and your detailed explanation. But could you please not do this anymore? It isn't malicious, but it's unhealthy for HN's ecosystem. For example, when an author submits his or her own work, that can add a lot of value to the community—but your bot pre-empts that, as it indeed did in this case. There are many more reasons why this isn't a good thing for HN. For example, it's better for submissions from popular sites to be distributed across a wide range of accounts. That gives more users a chance to feel like they're making important contributions, and gives the community (and authors) a clearer sense of the audience. There are lots of ways to write software to interact with HN, and lots of users with the ability to do it, so we really depend on the good will of the community only to do that when it serves the whole.

deleted 10y ago

[deleted]

Rapzid 10y ago

I've used firewall rules in the past to scope the metadata store to admin users.

cperciva 10y ago
That's better than nothing, but runs into problems if you have different users who need to be able to access different subsets of the metadata store.
- cesarb 10y ago
  An admin user could fetch these subsets of the metadata and leave a copy of them in the local filesystem.
  - cperciva 10y ago
    That doesn't work for IAM Roles, because (a) AWS library code expects to get the keys out of the metadata store, and (b) IAM Role credentials are periodically rotated, so the keys you downloaded in advance would expire.
    - Rapzid 10y ago
      You're right it doesn't. IAM roles are intended to be granted to the entire server, not a subset of the server's users. Any compromise of the server would be considered a compromise of its role. Yeah, this is a bit crazy depending on what you're running on it. I was running multi-tenant IIS hosts and the apps had no business with the metadata or ec2 IAM roles in my case. If you want roles to work for other users via the meta data store you can intercept requests with a proxy and then grab the temp credentials STS assume role. This is how kube2iam works. Depending on your use case you'd have to write the proxy, automate the mappings and firewall rules, etc etc etc. PITA but probably doable. On a different note, I agree with just about everything you had to say about the PITA that is IAM. Properly scoping permissions is much harder than it needs to be. Not all resources support tags, and even then almost nothing outside of ec2 supports tag conditions in IAM. This leads to many naming schemas and wildcard resource conditions :( AWS should have created the concept of resource groups. This would have greatly simplified giving users permissions to subsets of an accounts resources. I chalk this up to AWS's VERY poor collaboration between service teams. Nothing that came out seemed coordinated. This appears to be getting better (shrug).

0x0 10y ago

This is interesting! Can this be abused with AWS-hosted services that reach out to fetch URLs? For example, image hosts that allow specifying an URL to retrieve, or OAuth callbacks, etc? Are there any tricks to be played if someone were to register a random domain and point it to 169.254.169.254 (or worse, flux between 169.254.169.254 and a public IP in case there is blacklisting application code that first checks to resolve the hostname but then passes the whole URL into a library that resolves again?)

cperciva 10y ago
The EC2 documentation specifically warns against allowing yourself to be tricked into loading your metadata for someone else. So yes, there are probably lots of services which get this wrong; but they did at least get a warning about that particular failure mode.
idunno246 10y ago
Yes, one project I've worked on required a crawling bot, and you could crawl the metadata service...(until fixed). dont even need a domain in most cases, either the ip or instance-data DNS I bet works in a bunch of places. We redid all our policies to be extremely restrictive in response, if the instance did anything based on user input. Anything more admin happened on a different machine.
pfg 10y ago
That's a fairly common vulnerability. A good approach for services that need to fetch arbitrary URLs is roughly: 1. Resolve hostname and remember the response 2. Verify that the response does not contain any addresses in a private IP space, or any other IP that is only accessible to you 3. Use the IP from step 1 when establishing a connection With other solutions, you might end up being vulnerable to DNS rebinding attacks. Bonus points for doing all your URL fetching in some sort of sandbox that enforces these access rules.
- jamiesonbecker 10y ago
  This is accurate. Remember that even ELB's in AWS have IP's that change all the time, and this itself is actually a source of vulnerabilities from apps that don't respect DNS TTL's (as has been seen in the forums repeatedly -- apps get connected to the previous IP instead of the new one). It's probably safer to retrieve and verify the IP for each request, and just cache if the IP is 'safe'. (And just doing IP subnet calculations is non-trivial in most less-common languages.) Also, request throttling should be maintained and HTTP verb checking, to prevent being turned into a proxy for other attacks. Actually, any decision to accept an arbitrary URL should be carefully examined in light of how hard it is to do safely.

tex0 10y ago

It's much like the same Problem with the Google Cloud. Even worse there I'd say.

sgrytoyr 10y ago
Could you please elaborate? I'm not doubting you, just very interested in learning more.
boulos 10y ago
I just double checked, and the most similar thing we expose is the token's for each service-account in the instance metadata. As pointed out in the article, any uid on the box can read that. But, you can create instances with a zero-permission service account (the equivalent of nobody?) and just avoid it. This does mean that everywhere else you'd have to have explicit service accounts and such, but that seems like a reasonable "workaround" until or unless we make metadata access more granular (I like the block device idea! Would you want entirely different paths for JSON versus "plain" though?)
- lobster_johnson 10y ago
  Google Cloud does seem better here. The exception is GKE — Kubernetes nodes are associated with service accounts which have permissions that, if abused by a malicious Docker container, could be disasterous for your entire cluster. Considering the amount of unpatched Docker containers out there, that's a bit scary. It also effectively prevents GKE from being usable in any scenario where you want to schedule containers on behalf of third-party actors (think PaaS). (GKE also doesn't let you disable privileged Docker containers, but that's another story.) On AWS you can run a metadata proxy to prevent pods from getting the credentials, but I don't know of a clean way to accomplish the same thing on GKE.

yandie 10y ago

If you're sharing the same instance for multiple users, trying to achieve security among the users is almost impossible anyway. That's why physical separation/virtualization is one of the first thing to focus on when talking about security.

jamiesonbecker 10y ago
Isolation is definitely important, but not all parts of the system running a single function need the same levels of access, and in fact it may be possible to target those components separately. Take a look at the wikipedia articles for 'defense in depth' or 'privilege separation' to see how important it is inside a system to treat each component isolated to itself as much as possible. (This is also why you don't want to rely on only a perimeter firewall for access control.)

gtsteve 10y ago

This is an interesting attack that I must confess I hadn't thought of, but surely any service that accepts an arbitrary URL has a list of IP ranges to avoid. However, to harden a role in the event of instance role credentials leaking, you could use an IAM Condition [0]. There is actually an example of this in the IAM documentation [1], although the source VPC parameter doesn't work for all services, and I can't see a list of services that support this parameter. This would ensure that the requests actually came from instances within your VPC. [0] http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html#Condition http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_po... [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies-vpc-endpoint.html#example-bucket-policies-restrict-access-vpc http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucke...

jamiesonbecker 10y ago
The point is not requests that originate elsewhere. The point is that this system is not protected in any way from any other process on your system.
BraveNewCurency 10y ago
Er, the problem is not "people can hit an un-routable IP from outside your instance". The problem is that "if your instance allows an attacker to make a HTTP request, you might expose personal information". For example, web crawlers or other fetchers.

logronoide 10y ago

Same happens with metadata access in Openstack. The access is controlled by source IP (and namespace). I wonder if it's possible to spoof the IP and access Metadata of other servers/users.

djb_hackernews 10y ago

If users can issue arbitrary commands on an instance then that instance should have zero Iam roles and should delegate actions to services running on separate instances. The instances hosting our users go a step further and null route Metadata service requests via iptables.

jeremyjh 10y ago
It isn't just about users, its also about malicious software you may accidentally install, if for example a library you use is compromised as has happened before with Ruby gems.

cesarb 10y ago

What I've done for a previous company was to, as one of the very first things done within every EC2 instance, add an iptables owner match rule to only allow packets destined to 169.254.169.254 if they come from uid 0. Any information from that webservice that non-root users might need (for instance, the EC2 instance ID) is fetched on boot by a script running as root and left somewhere in the filesystem.

falcolas 10y ago
This won't help with IAM roles, since the credentials provided in the metadata expire. Of course, a small tweak to the iptables entry would help there as well. Mind posting your entry for us iptables impared folks?
- cesarb 10y ago
  I don't work there anymore, so I don't have access to the exact rule I used, but IIRC it was something like iptables -t filter -I OUTPUT -d 169.254.169.254 -m owner \! --uid-owner 0 -j REJECT --reject-with icmp-admin-prohibited
jeremyjh 10y ago
That is the obvious answer, do you have any scripts you could share?
manojlds 10y ago
But the data is not static.
- abofh 10y ago
  Most of it is - your instance ID, network position, etc, will never change after boot (or 'start' if you stop it) - so caching it is just fine. There's very little that will change on a running instance except of course, the IAM credentials referred to in this article (as they expire within 90 minutes IIRC).
  - 10y ago

patsplat 10y ago

EC2 instances are designed to enforce isolation between instances, not processes. Presumably there would only be one primary service running on each. Use AWS be pushed towards an architecture based on containers and services. AWS is the OS, not any individual machine.

hueving 10y ago

The blog post buries the lead a little bit because it's talking about lots of pain points with the ec2 API and IAM. The important point to take away is that any process with network access running on your instance can contact the EC2 metadata service at http://169.254.169.254 http://169.254.169.254 and get the instance-specific IAM credentials. Think about things like services that accept user submitted URLs, crawl them, and display results...

kevsim 10y ago
I believe Pocket faced exactly this issue once upon a time.
- Qerub 10y ago
  Indeed! https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-pocket/ https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-po...
strictfp 10y ago
....and that you can imitate the metadata service to make life easier :) A plug for my friends side project: https://github.com/otm/limes https://github.com/otm/limes . It's a local metadataservice. Very handy for making aws libs work without having to config them much. And great support for MFAs.
deleted 10y ago
[deleted]
novaleaf 10y ago
do you know if any similar vulnerability exists with Azure or Google Cloud? edit: not sure why the downvote, I use Google Cloud, so honest question :(
- Pharaoh2 10y ago
  You are getting downvoted because it's not a vulnerability. It's well documented and very useful feature in aws and it also exists in gcloud, although gcloud documentation is not as good^1. It only becomes a vulnerability if you don't read the documentation AND don't follow best practices^2. I don't know if azure provides this feature. ^1 I use gcloud now and have used aws in the past. ^2 Don't run unknown code without proper sandboxing

rcaught 10y ago

> almost as trivial for EC2 instances to expose XenStore as a filesystem to which standard UNIX permissions could be applied, providing IAM Role credentials with the full range of access control functionality which UNIX affords to files stored on disk. Doesn't this become more complicated when you think about EC2 offering Windows instances? Even with straight UNIX file writing, what writes this? Where does it write this? Which user has read permissions?

skywhopper 10y ago
Yeah, having the metadata available over an http interface is actually brilliant. Simple HTTP calls are easy to do from any network-capable OS or language.
- jamiesonbecker 10y ago
  So is reading a file on the filesystem.. easier, actually in most languages, since HTTP requests usually require loading an extra library.
  - ChoHag 10y ago
    A library which uses the same (built-in) underlying IO mechanisms as the (built-in) filesystem.
jamiesonbecker 10y ago
In UNIX, the same way that EBS volumes are mounted... think of the /proc or /sys virtual filesystems. In Windows, I'm guessing that this would be exposed as a network drive.
- rcaught 10y ago
  My point is that these type of solutions create a lot more overhead, inconsistency and variation compared to a HTTP request; granted, less security.

skywhopper 10y ago

Hopefully the operators using EC2 instance profiles understand and weigh the risks of using that feature. It's good to be cautious, but the feature is only dangerous if you don't take the time to understand it. Running a server on the Internet at all is "dangerous" in the same sense. And for this particular risk, it turns out there's a simple fix. He _is_ right in his first criticism that the IAM access controls available for much of the AWS API are entirely inadequate. In the case of EC2 in particular, it's all or nothing--either your credentials can call the TerminateInstances API or they can't. I'm sure Amazon is working on improving things, but for now it's pretty terrible. But in practice it just means you have to take care in different ways than you would if his tag-based authz solution were implemented. That said, while it's certainly frustrating to an implementor, it's not "dangerous" that limitations exist in these APIs. We're talking about decade-old APIs from the earliest days of AWS, and while things have been added, the core APIs are still the same. That's an amazing success story. But like any piece of software, there are issues that experienced users learn how to work around. You can bet that the EC2 API code is hard and scary to deal with for its maintainers. Adding a huge new permissions scheme is likely nearly impossible without a total rewrite... I don't envy them their task.

jamiesonbecker 10y ago
It's impossible to limit access to any part of the instance metadata in any way w/o firewalling (which has its own issues) or even to expire access to any part of it. Since instance profiles have keys (even though automatically rotated), any process on the system, owned by any user, can access anything exposed via the instance role. This makes embedding IAM keys into your instance and protecting it by root-only or ACL's MUCH MUCH safer... but AWS specifically states that instance profiles are preferred. In fact, for our Userify AWS instances (ssh key management), we are required to use instance roles and not allowed to offer the option. (This is why we do not offer S3 bucket storage on our AWS instances but we do on Pro and Enterprise self-hosted.) The biggest issue with the IAM instance profiles is that they trade security for convenience.. and it's not a good trade.
- subway 10y ago
  For the most part EC2 instances should be single-purpose. Use tiny instances that do one job. Your IAM role describes the permissions that should be granted to that one job. It's absolutely true that you cannot isolate permissions at the process level, but by using single-job-type instances, you can easily isolate permissions on a per-job (in this model, per-instance) basis.

_hyn3 10y ago

tl;dr: 1) IAM instance roles have no security mechanisms to protect them from being read from any process on the instance, thus completely eliminating them from all Linux/UNIX/Windows permission systems. (The real reason for this is that instance metadata was a convenient semi-public information store for things like instance ID, but it was extended to also provide secret material, which was, at best, an idiotic move.) As the author points out, Xen already provided a great filesystem alternative that could be mounted as another drive (or network drive) to be managed with the regular OS filesystem permission system. (reading an instance ID is just a matter of reading a "file")... for some reason, AWS didn't leverage this and instead just added the secret material to its local instance metadata webserver. 2) the API calls are not fine grained enough and/or there are big holes in their coverage -- so, for instance, if you want to use some other AWS services, you can end up exposing much more than you intended.

deleted 10y ago

[deleted]

jamiesonbecker 10y ago

Until AWS fixes this (which, as the article points out, may never happen), a chmod'ed 600 file (only readable by root) is actually much safer, even when STS auto-rotation is taken into account.

icedchai 10y ago

IAM instance roles are still an improvement over how it was typically done in the past: hard-coding the same key in a configuration file and deploying it everywhere. It's a balance between security and convenience.

ChoHag 10y ago

The number of people in this thread not merely nodding their heads and mmhmm-ing (or the internet equivalent) is a concern.

zimbatm 10y ago

I wonder how many services on Amazon allow user-configurable webhooks that can be pointed to http://169.254.169.254 http://169.254.169.254 ...

Thaxll 10y ago

It has been the case for 10 years, anyone knows that, I don't see the problem. If you're not happy with it just use API keys.

helper 10y ago
IAM instance roles were only added in 2012: https://aws.amazon.com/blogs/aws/iam-roles-for-ec2-instances-simplified-secure-access-to-aws-service-apis-from-ec2/ https://aws.amazon.com/blogs/aws/iam-roles-for-ec2-instances...