5 ms·

Overview of all Amazon AWS APIs

171 points by nl5887 10y ago

_ao789 10y ago
Very nice. It would be cool if you could try out api calls from the examples (ala swagger style)
- TheDong 10y ago
  Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account. Providing an AWS account that people could use is a security nightmare, and many of the operations have charges associated with them. If I provide my aws credentials, then he could proxy through his server and request on my behalf, but I wouldn't want to give him my credentials and I might as well just play with the aws-cli which exposes pretty much all of the same calls anyways.
  - pqhwan 10y ago
    Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.
    - derefr 10y ago
      I was considering writing something similar: a "semi-mocked" implementation of all of AWS, getting you most of what you get from running OpenStack but from one static binary running in userland and with no persistence. My use-case for it was that I'd write it all in Erlang, and then each public port on a mocked EC2 instance would have a single Erlang process hooked up to it to respond to messages coming to that instance-IP:port. So you could set up something that looked like a thousand-node distributed VPC, from the perspective of an AWS client talking to the API, and it would all sit in a couple MB of RAM and near-zero CPU.
    - vincentdm 10y ago
      This! A local mock of AWS is sorely lacking for most services. The best one I know are Amazon's own DynamoDB and the independent FakeS3 tool which exposes an http api and saves the blobs and metadata as local files in a transparent way.
    - jonaf 10y ago
      I've been waiting for someone to come along and say "I did this!" Okay so I didn't do this, but I have seen several folks where I work create some stubbed AWS services to facilitate testing. I'll have to ask them if this turned out to be worth the (gargantuan) effort. Edit: Looks like there are a couple[1] instances[2] where people have done this to some degree. Many of them are very old, though. The one built by a team where I work only implements SNS/SQS as well, so there's a pretty big gap for other interfaces. It seems like the recommended way to do this is to just use something like EasyMock or whatever and stub rather than have an actual API implemented for integration testing. I'm not sure I'm satisfied with that answer; on the other hand, it's substantially easy for a custom implementation to wildly differ from the actual AWS API behavior... Seems risky to base integration testing results on that? [1] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [2] https://github.com/unbounce/yopa https://github.com/unbounce/yopa
    - niftich 10y ago
      Here's some third-party implementations of some AWS APIs, with no endorsement of quality: S3 [A1] https://github.com/jubos/fake-s3 https://github.com/jubos/fake-s3 [A2] https://github.com/scality/s3 https://github.com/scality/s3 [A3] http://s3ninja.net/ http://s3ninja.net/ [A4] https://github.com/basho/riak_cs https://github.com/basho/riak_cs [A5] https://www.npmjs.com/package/s3rver https://www.npmjs.com/package/s3rver [A6] http://ceph.com/ceph-storage/object-storage/ http://ceph.com/ceph-storage/object-storage/ ---- SQS / SNS [B1] https://github.com/iain/fake_sqs https://github.com/iain/fake_sqs [B2] https://github.com/yourkarma/fake_sns https://github.com/yourkarma/fake_sns [B3] https://github.com/adamw/elasticmq https://github.com/adamw/elasticmq [B4] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [B5] http://stratosphere.codeplex.com/ http://stratosphere.codeplex.com/ [B6] https://github.com/unbounce/yopa https://github.com/unbounce/yopa ---- DynamoDB [C1] https://github.com/mhart/dynalite https://github.com/mhart/dynalite [C2] https://github.com/ananthakumaran/fake_dynamo https://github.com/ananthakumaran/fake_dynamo ---- SimpleDB [D1] https://github.com/stephenh/fakesdb https://github.com/stephenh/fakesdb
      - TheAceOfHearts 10y ago
        I can't speak for all the libraries linked above, but I've used fake-s3 with a rails app and s3rver with a node app. They both worked great for development and testing. I'd definitely use em again. I think it's worth noting that some of these libraries have different intended goals. For example: Riak CS is intended as an alternative that you can use in production, while s3rver is meant for testing.
      - y4m4b4 10y ago
        Another project to be added to this list would be https://github.com/minio/minio https://github.com/minio/minio - Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License. Disclaimer: I work for Minio.
  - cavisne 10y ago
    +1 on just using the CLI but could also use STS to generate a token with a locked down policy client side and the risk of proxying would be much smaller
  - dastbe 10y ago
    "Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account." The first part is rapidly becoming less true; any of the services enabled by default on this page support browser calls by default. https://sdk.amazonaws.com/builder/js/# https://sdk.amazonaws.com/builder/js/#
  - jonaf 10y ago
    There was a similar HN post like this but for Elasticsearch a few weeks ago. I'm kind of thinking that a command line tool for elastic search similar to the AWS command line isn't a bad idea. In general when I can't remember an AWS API call, instead of googling, I just use the AWS cli. I find it faster than google or a reference like this. Anyone else?
dkarapetyan 10y ago
Keeps redirecting to S3 APIs for me. Is that intended? One thing people should know about AWS and AWS APIs in general is that it is a ghetto. The ad-hoc nature of most AWS services and their weird interactions is indicative of generally bad design. Even with the AWS Ruby SDK I can barely get anything done without consulting 5 different references about which parameters are required, which are optional, and what the sequence of various calls is supposed to be to get an intended result. So even though this is useful a cookbook would have been much more useful.
- Daviey 10y ago
  Worse, when I worked on OpenStack's EC2 compatibility API.. I found about 3 examples where AWS were not following their own published API parameters... and I had to choose to follow either their published spec, or what they were actually doing.
- percept 10y ago
  I don't know if it's bad, or just rapidly evolving. Somebody should create a cool map/visualization of the AWS landscape. (Possibly relating it to some other concept: AWS as a small town, Disney World, or Simpsons' Springfield?)
thisismyhnuser 10y ago
I really wish someone would re-write all of AWS documentation and make everything simpler to understand. I'd like to use AWS but the documentation as-is would take hours to read.
- pg_is_a_butt 10y ago
  i've had to do large complex integrations with tons of cloud service providers... AWS has way better docs than most. i can assure you it would take more than a few hours to deal with the issues that would come up if they left any of the details out, and you had to figure them out the hard way while they bit you in the ass. twilio is another example of good docs.
- true_religion 10y ago
  AWS docs, at least with S3, are pretty rigorous which makes implementing an S3 client directly from the documentation possible. I'm sure they expect you to use a client written in your language of choice, not actually send queries to the API manually.
- fideloper 10y ago
  Within each SDK, I find their API relatively easy to use. Each method is pretty well documented (certainly not perfectly). For the most part they are created via code generation (to my knowledge), which while not necessarily making for the best user-friendly API (Although it's not bad), does make it fairly consistent in terms of groking how to accomplish automating a task.
- michaelbuckbee 10y ago
  It's not everything - just an overview - but you might find this helpful: https://www.expeditedssl.com/aws-in-plain-english https://www.expeditedssl.com/aws-in-plain-english
  - kchoudhu 10y ago
    Has this been edited since it was originally posted? I recall reading something about setting money on fire in the one version back in the day...
TheDong 10y ago
What does this provide compared to the official documentation for each service available on Amazon's website, e.g. https://docs.aws.amazon.com/amazonswf/latest/apireference/API_DescribeActivityType.html https://docs.aws.amazon.com/amazonswf/latest/apireference/AP... Each one is available from https://aws.amazon.com/documentation/ https://aws.amazon.com/documentation/ -> click service -> click "Api Reference" If all you've saved is one click, I don't think it's worth it, so what else does this do?
- nl5887 10y ago
  I've created this as my quick and easy reference to all Amazon api's. This works for me better to browse quickly between different calls, and suits my personal workflow more.
posnet 10y ago
Which one of the AWS SDKs is this based on? Or did you actually scrape the documentation web pages?
- nl5887 10y ago
  It is based on the json reference of the aws api, which is being used to generated all different SDKs.
karavelov 10y ago
The whole AWS APIs are formalized and exposed in different ways and that formalism is one of it's strengths: everybody can write a transformation that creates a bindings for his language of choice. For example of this take a look at: https://github.com/boto/boto3/tree/develop/boto3/data https://github.com/boto/boto3/tree/develop/boto3/data P.S. Agree that not all of the services adhere to the same bar.
Tillie95 10y ago
yaya

171 points by nl5887 10y ago

_ao789 10y ago

Very nice. It would be cool if you could try out api calls from the examples (ala swagger style)

TheDong 10y ago
Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account. Providing an AWS account that people could use is a security nightmare, and many of the operations have charges associated with them. If I provide my aws credentials, then he could proxy through his server and request on my behalf, but I wouldn't want to give him my credentials and I might as well just play with the aws-cli which exposes pretty much all of the same calls anyways.
- pqhwan 10y ago
  Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.
  - derefr 10y ago
    I was considering writing something similar: a "semi-mocked" implementation of all of AWS, getting you most of what you get from running OpenStack but from one static binary running in userland and with no persistence. My use-case for it was that I'd write it all in Erlang, and then each public port on a mocked EC2 instance would have a single Erlang process hooked up to it to respond to messages coming to that instance-IP:port. So you could set up something that looked like a thousand-node distributed VPC, from the perspective of an AWS client talking to the API, and it would all sit in a couple MB of RAM and near-zero CPU.
  - vincentdm 10y ago
    This! A local mock of AWS is sorely lacking for most services. The best one I know are Amazon's own DynamoDB and the independent FakeS3 tool which exposes an http api and saves the blobs and metadata as local files in a transparent way.
  - jonaf 10y ago
    I've been waiting for someone to come along and say "I did this!" Okay so I didn't do this, but I have seen several folks where I work create some stubbed AWS services to facilitate testing. I'll have to ask them if this turned out to be worth the (gargantuan) effort. Edit: Looks like there are a couple[1] instances[2] where people have done this to some degree. Many of them are very old, though. The one built by a team where I work only implements SNS/SQS as well, so there's a pretty big gap for other interfaces. It seems like the recommended way to do this is to just use something like EasyMock or whatever and stub rather than have an actual API implemented for integration testing. I'm not sure I'm satisfied with that answer; on the other hand, it's substantially easy for a custom implementation to wildly differ from the actual AWS API behavior... Seems risky to base integration testing results on that? [1] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [2] https://github.com/unbounce/yopa https://github.com/unbounce/yopa
  - niftich 10y ago
    Here's some third-party implementations of some AWS APIs, with no endorsement of quality: S3 [A1] https://github.com/jubos/fake-s3 https://github.com/jubos/fake-s3 [A2] https://github.com/scality/s3 https://github.com/scality/s3 [A3] http://s3ninja.net/ http://s3ninja.net/ [A4] https://github.com/basho/riak_cs https://github.com/basho/riak_cs [A5] https://www.npmjs.com/package/s3rver https://www.npmjs.com/package/s3rver [A6] http://ceph.com/ceph-storage/object-storage/ http://ceph.com/ceph-storage/object-storage/ ---- SQS / SNS [B1] https://github.com/iain/fake_sqs https://github.com/iain/fake_sqs [B2] https://github.com/yourkarma/fake_sns https://github.com/yourkarma/fake_sns [B3] https://github.com/adamw/elasticmq https://github.com/adamw/elasticmq [B4] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [B5] http://stratosphere.codeplex.com/ http://stratosphere.codeplex.com/ [B6] https://github.com/unbounce/yopa https://github.com/unbounce/yopa ---- DynamoDB [C1] https://github.com/mhart/dynalite https://github.com/mhart/dynalite [C2] https://github.com/ananthakumaran/fake_dynamo https://github.com/ananthakumaran/fake_dynamo ---- SimpleDB [D1] https://github.com/stephenh/fakesdb https://github.com/stephenh/fakesdb
    - TheAceOfHearts 10y ago
      I can't speak for all the libraries linked above, but I've used fake-s3 with a rails app and s3rver with a node app. They both worked great for development and testing. I'd definitely use em again. I think it's worth noting that some of these libraries have different intended goals. For example: Riak CS is intended as an alternative that you can use in production, while s3rver is meant for testing.
    - y4m4b4 10y ago
      Another project to be added to this list would be https://github.com/minio/minio https://github.com/minio/minio - Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License. Disclaimer: I work for Minio.
- cavisne 10y ago
  +1 on just using the CLI but could also use STS to generate a token with a locked down policy client side and the risk of proxying would be much smaller
- dastbe 10y ago
  "Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account." The first part is rapidly becoming less true; any of the services enabled by default on this page support browser calls by default. https://sdk.amazonaws.com/builder/js/# https://sdk.amazonaws.com/builder/js/#
- jonaf 10y ago
  There was a similar HN post like this but for Elasticsearch a few weeks ago. I'm kind of thinking that a command line tool for elastic search similar to the AWS command line isn't a bad idea. In general when I can't remember an AWS API call, instead of googling, I just use the AWS cli. I find it faster than google or a reference like this. Anyone else?

dkarapetyan 10y ago

Keeps redirecting to S3 APIs for me. Is that intended? One thing people should know about AWS and AWS APIs in general is that it is a ghetto. The ad-hoc nature of most AWS services and their weird interactions is indicative of generally bad design. Even with the AWS Ruby SDK I can barely get anything done without consulting 5 different references about which parameters are required, which are optional, and what the sequence of various calls is supposed to be to get an intended result. So even though this is useful a cookbook would have been much more useful.

Daviey 10y ago
Worse, when I worked on OpenStack's EC2 compatibility API.. I found about 3 examples where AWS were not following their own published API parameters... and I had to choose to follow either their published spec, or what they were actually doing.
percept 10y ago
I don't know if it's bad, or just rapidly evolving. Somebody should create a cool map/visualization of the AWS landscape. (Possibly relating it to some other concept: AWS as a small town, Disney World, or Simpsons' Springfield?)

thisismyhnuser 10y ago

I really wish someone would re-write all of AWS documentation and make everything simpler to understand. I'd like to use AWS but the documentation as-is would take hours to read.

pg_is_a_butt 10y ago
i've had to do large complex integrations with tons of cloud service providers... AWS has way better docs than most. i can assure you it would take more than a few hours to deal with the issues that would come up if they left any of the details out, and you had to figure them out the hard way while they bit you in the ass. twilio is another example of good docs.
true_religion 10y ago
AWS docs, at least with S3, are pretty rigorous which makes implementing an S3 client directly from the documentation possible. I'm sure they expect you to use a client written in your language of choice, not actually send queries to the API manually.
fideloper 10y ago
Within each SDK, I find their API relatively easy to use. Each method is pretty well documented (certainly not perfectly). For the most part they are created via code generation (to my knowledge), which while not necessarily making for the best user-friendly API (Although it's not bad), does make it fairly consistent in terms of groking how to accomplish automating a task.
michaelbuckbee 10y ago
It's not everything - just an overview - but you might find this helpful: https://www.expeditedssl.com/aws-in-plain-english https://www.expeditedssl.com/aws-in-plain-english
- kchoudhu 10y ago
  Has this been edited since it was originally posted? I recall reading something about setting money on fire in the one version back in the day...

TheDong 10y ago

What does this provide compared to the official documentation for each service available on Amazon's website, e.g. https://docs.aws.amazon.com/amazonswf/latest/apireference/API_DescribeActivityType.html https://docs.aws.amazon.com/amazonswf/latest/apireference/AP... Each one is available from https://aws.amazon.com/documentation/ https://aws.amazon.com/documentation/ -> click service -> click "Api Reference" If all you've saved is one click, I don't think it's worth it, so what else does this do?

nl5887 10y ago
I've created this as my quick and easy reference to all Amazon api's. This works for me better to browse quickly between different calls, and suits my personal workflow more.

posnet 10y ago

Which one of the AWS SDKs is this based on? Or did you actually scrape the documentation web pages?

nl5887 10y ago
It is based on the json reference of the aws api, which is being used to generated all different SDKs.

karavelov 10y ago

The whole AWS APIs are formalized and exposed in different ways and that formalism is one of it's strengths: everybody can write a transformation that creates a bindings for his language of choice. For example of this take a look at: https://github.com/boto/boto3/tree/develop/boto3/data https://github.com/boto/boto3/tree/develop/boto3/data P.S. Agree that not all of the services adhere to the same bar.

Tillie95 10y ago

yaya