4 ms·

Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.

by pqhwan 10y ago

derefr 10y ago
I was considering writing something similar: a "semi-mocked" implementation of all of AWS, getting you most of what you get from running OpenStack but from one static binary running in userland and with no persistence. My use-case for it was that I'd write it all in Erlang, and then each public port on a mocked EC2 instance would have a single Erlang process hooked up to it to respond to messages coming to that instance-IP:port. So you could set up something that looked like a thousand-node distributed VPC, from the perspective of an AWS client talking to the API, and it would all sit in a couple MB of RAM and near-zero CPU.
vincentdm 10y ago
This! A local mock of AWS is sorely lacking for most services. The best one I know are Amazon's own DynamoDB and the independent FakeS3 tool which exposes an http api and saves the blobs and metadata as local files in a transparent way.
jonaf 10y ago
I've been waiting for someone to come along and say "I did this!" Okay so I didn't do this, but I have seen several folks where I work create some stubbed AWS services to facilitate testing. I'll have to ask them if this turned out to be worth the (gargantuan) effort. Edit: Looks like there are a couple[1] instances[2] where people have done this to some degree. Many of them are very old, though. The one built by a team where I work only implements SNS/SQS as well, so there's a pretty big gap for other interfaces. It seems like the recommended way to do this is to just use something like EasyMock or whatever and stub rather than have an actual API implemented for integration testing. I'm not sure I'm satisfied with that answer; on the other hand, it's substantially easy for a custom implementation to wildly differ from the actual AWS API behavior... Seems risky to base integration testing results on that? [1] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [2] https://github.com/unbounce/yopa https://github.com/unbounce/yopa
niftich 10y ago
Here's some third-party implementations of some AWS APIs, with no endorsement of quality: S3 [A1] https://github.com/jubos/fake-s3 https://github.com/jubos/fake-s3 [A2] https://github.com/scality/s3 https://github.com/scality/s3 [A3] http://s3ninja.net/ http://s3ninja.net/ [A4] https://github.com/basho/riak_cs https://github.com/basho/riak_cs [A5] https://www.npmjs.com/package/s3rver https://www.npmjs.com/package/s3rver [A6] http://ceph.com/ceph-storage/object-storage/ http://ceph.com/ceph-storage/object-storage/ ---- SQS / SNS [B1] https://github.com/iain/fake_sqs https://github.com/iain/fake_sqs [B2] https://github.com/yourkarma/fake_sns https://github.com/yourkarma/fake_sns [B3] https://github.com/adamw/elasticmq https://github.com/adamw/elasticmq [B4] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [B5] http://stratosphere.codeplex.com/ http://stratosphere.codeplex.com/ [B6] https://github.com/unbounce/yopa https://github.com/unbounce/yopa ---- DynamoDB [C1] https://github.com/mhart/dynalite https://github.com/mhart/dynalite [C2] https://github.com/ananthakumaran/fake_dynamo https://github.com/ananthakumaran/fake_dynamo ---- SimpleDB [D1] https://github.com/stephenh/fakesdb https://github.com/stephenh/fakesdb
- TheAceOfHearts 10y ago
  I can't speak for all the libraries linked above, but I've used fake-s3 with a rails app and s3rver with a node app. They both worked great for development and testing. I'd definitely use em again. I think it's worth noting that some of these libraries have different intended goals. For example: Riak CS is intended as an alternative that you can use in production, while s3rver is meant for testing.
- y4m4b4 10y ago
  Another project to be added to this list would be https://github.com/minio/minio https://github.com/minio/minio - Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License. Disclaimer: I work for Minio.

Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.

by pqhwan 10y ago

Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.

I was considering writing something similar: a "semi-mocked" implementation of all of AWS, getting you most of what you get from running OpenStack but from one static binary running in userland and with no persistence. My use-case for it was that I'd write it all in Erlang, and then each public port on a mocked EC2 instance would have a single Erlang process hooked up to it to respond to messages coming to that instance-IP:port. So you could set up something that looked like a thousand-node distributed VPC, from the perspective of an AWS client talking to the API, and it would all sit in a couple MB of RAM and near-zero CPU.

vincentdm 10y ago

This! A local mock of AWS is sorely lacking for most services. The best one I know are Amazon's own DynamoDB and the independent FakeS3 tool which exposes an http api and saves the blobs and metadata as local files in a transparent way.

jonaf 10y ago

I've been waiting for someone to come along and say "I did this!" Okay so I didn't do this, but I have seen several folks where I work create some stubbed AWS services to facilitate testing. I'll have to ask them if this turned out to be worth the (gargantuan) effort. Edit: Looks like there are a couple[1] instances[2] where people have done this to some degree. Many of them are very old, though. The one built by a team where I work only implements SNS/SQS as well, so there's a pretty big gap for other interfaces. It seems like the recommended way to do this is to just use something like EasyMock or whatever and stub rather than have an actual API implemented for integration testing. I'm not sure I'm satisfied with that answer; on the other hand, it's substantially easy for a custom implementation to wildly differ from the actual AWS API behavior... Seems risky to base integration testing results on that? [1] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [2] https://github.com/unbounce/yopa https://github.com/unbounce/yopa

niftich 10y ago

Here's some third-party implementations of some AWS APIs, with no endorsement of quality: S3 [A1] https://github.com/jubos/fake-s3 https://github.com/jubos/fake-s3 [A2] https://github.com/scality/s3 https://github.com/scality/s3 [A3] http://s3ninja.net/ http://s3ninja.net/ [A4] https://github.com/basho/riak_cs https://github.com/basho/riak_cs [A5] https://www.npmjs.com/package/s3rver https://www.npmjs.com/package/s3rver [A6] http://ceph.com/ceph-storage/object-storage/ http://ceph.com/ceph-storage/object-storage/ ---- SQS / SNS [B1] https://github.com/iain/fake_sqs https://github.com/iain/fake_sqs [B2] https://github.com/yourkarma/fake_sns https://github.com/yourkarma/fake_sns [B3] https://github.com/adamw/elasticmq https://github.com/adamw/elasticmq [B4] https://github.com/p4tin/goaws https://github.com/p4tin/goaws [B5] http://stratosphere.codeplex.com/ http://stratosphere.codeplex.com/ [B6] https://github.com/unbounce/yopa https://github.com/unbounce/yopa ---- DynamoDB [C1] https://github.com/mhart/dynalite https://github.com/mhart/dynalite [C2] https://github.com/ananthakumaran/fake_dynamo https://github.com/ananthakumaran/fake_dynamo ---- SimpleDB [D1] https://github.com/stephenh/fakesdb https://github.com/stephenh/fakesdb

TheAceOfHearts 10y ago
I can't speak for all the libraries linked above, but I've used fake-s3 with a rails app and s3rver with a node app. They both worked great for development and testing. I'd definitely use em again. I think it's worth noting that some of these libraries have different intended goals. For example: Riak CS is intended as an alternative that you can use in production, while s3rver is meant for testing.
y4m4b4 10y ago
Another project to be added to this list would be https://github.com/minio/minio https://github.com/minio/minio - Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License. Disclaimer: I work for Minio.