5 ms·

Event Notifications for Amazon S3

61 points by degio 12y ago

mathgladiator 12y ago
Of all the features that S3 has shipped, this is by far my favorite. I look forward to utilizing Lambda and S3 to build an ultra cheap content management service to drive the cost of my website management to less than a $1/mo.
- toomuchtodo 12y ago
  You could definitely kill shared hosting services (with static sites) with this sort of offering. They could never compete at S3 pricing.
- kolev 12y ago
  How is this idea any better (and cheaper) than a static site generator like Jekyll?
dozy 12y ago
I was very excited when I saw this headline, as for a while I've hosted various things on s3 and have wanted an easy way to track when/how/by whom those things have been downloaded. Alas, that isn't a supported...I'm looking for something like: "s3:ObjectViewed:Get". Or am I mistaken and that already exists in some other form...?
- mritun 12y ago
  S3 already has a feature where you can configure per-bucket apache style access logs which log every operation and may solve your purpose. This feature provides realtime messaging for uploads.
  - dozy 12y ago
    I'll take it! Good to know.
johne20 12y ago
At first glance, this combined with Lambda would make for an awesome server-less image transformation setup.
strick 12y ago
Have I missed something or is there no way to get notified when an object gets deleted?
- zwily 12y ago
  Not yet.
Yadi 12y ago
This is awesome, and the cost is pretty affordable. I don't know if the lambada feature was there before, but it's pretty cool!
- mathgladiator 12y ago
  Lambda was announced today: https://news.ycombinator.com/item?id=8602936 https://news.ycombinator.com/item?id=8602936
hemancuso 12y ago
Very cool! Are there any ordering guarantees for these notifications? Specifically the order in which objects were created? It would be nice to be able to generate a meaningful event log.
deleted 12y ago
[deleted]
- mathgladiator 12y ago
  The AWS conference Re:Invent is happening right now: http://reinvent.awsevents.com/live/breakouts.html http://reinvent.awsevents.com/live/breakouts.html
skynss 12y ago
Too bad they did not differentiate between Create and Overwrite (as far as I can tell) as that is a an important difference (I need to solve scenarios where idempotency is needed - more than 1 worker may create exact same email message due to fault tolerance design, but at least AND at most 1 email should be sent to customer. I was hoping that the s3 filename/key would be the unique identifier and if there was separate notification for overwrite than from create, email could be sent only for create event only.)
- jeffbarr 12y ago
  You could use S3 versioning to distinguish the two operations.
  - kolev 12y ago
    S3 versioning could be pricey and painful. Try deleting a versioned bucket! I wrote a tool that I'm gonna open-source if my employer okays it, but the pathetic part is that you get charged for each HTTP request - imagine a bucket with a billion objects and a gazillion versions!
    - hemancuso 12y ago
      http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectde...
      - kolev 12y ago
        That's the API I use as well. The DELETEs are free, but GETs, PUTs, POSTs, and LISTs are not [1]. [1] http://aws.amazon.com/s3/pricing/ http://aws.amazon.com/s3/pricing/
    - evansd 12y ago
      If you're not in a huge hurry to delete the bucket you can easily set lifecycle rules to expire all versions older than 1 day (the minimum) and then just wait for S3 to delete them for you.
kolev 12y ago
Nice feature, but, again, not available in CloudFormation. I understand the challenge to coordinate releases, I can live with a week or two lag, but usually it's months - CloudFormation still doesn't support basic AWS services. I really am considering abandoning this gem, which is highly neglected and underutilized. Terraform [1] looks like something I can extend myself easily if the upstream doesn't support something just yet and contribute it as a pull request. [1] https://terraform.io/ https://terraform.io/

61 points by degio 12y ago

Of all the features that S3 has shipped, this is by far my favorite. I look forward to utilizing Lambda and S3 to build an ultra cheap content management service to drive the cost of my website management to less than a $1/mo.

toomuchtodo 12y ago
You could definitely kill shared hosting services (with static sites) with this sort of offering. They could never compete at S3 pricing.
kolev 12y ago
How is this idea any better (and cheaper) than a static site generator like Jekyll?

dozy 12y ago

I was very excited when I saw this headline, as for a while I've hosted various things on s3 and have wanted an easy way to track when/how/by whom those things have been downloaded. Alas, that isn't a supported...I'm looking for something like: "s3:ObjectViewed:Get". Or am I mistaken and that already exists in some other form...?

mritun 12y ago
S3 already has a feature where you can configure per-bucket apache style access logs which log every operation and may solve your purpose. This feature provides realtime messaging for uploads.
- dozy 12y ago
  I'll take it! Good to know.

johne20 12y ago

At first glance, this combined with Lambda would make for an awesome server-less image transformation setup.

strick 12y ago

Have I missed something or is there no way to get notified when an object gets deleted?

zwily 12y ago
Not yet.

Yadi 12y ago

This is awesome, and the cost is pretty affordable. I don't know if the lambada feature was there before, but it's pretty cool!

mathgladiator 12y ago
Lambda was announced today: https://news.ycombinator.com/item?id=8602936 https://news.ycombinator.com/item?id=8602936

hemancuso 12y ago

Very cool! Are there any ordering guarantees for these notifications? Specifically the order in which objects were created? It would be nice to be able to generate a meaningful event log.

deleted 12y ago

[deleted]

mathgladiator 12y ago
The AWS conference Re:Invent is happening right now: http://reinvent.awsevents.com/live/breakouts.html http://reinvent.awsevents.com/live/breakouts.html

skynss 12y ago

Too bad they did not differentiate between Create and Overwrite (as far as I can tell) as that is a an important difference (I need to solve scenarios where idempotency is needed - more than 1 worker may create exact same email message due to fault tolerance design, but at least AND at most 1 email should be sent to customer. I was hoping that the s3 filename/key would be the unique identifier and if there was separate notification for overwrite than from create, email could be sent only for create event only.)

jeffbarr 12y ago
You could use S3 versioning to distinguish the two operations.
- kolev 12y ago
  S3 versioning could be pricey and painful. Try deleting a versioned bucket! I wrote a tool that I'm gonna open-source if my employer okays it, but the pathetic part is that you get charged for each HTTP request - imagine a bucket with a billion objects and a gazillion versions!
  - hemancuso 12y ago
    http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectde...
    - kolev 12y ago
      That's the API I use as well. The DELETEs are free, but GETs, PUTs, POSTs, and LISTs are not [1]. [1] http://aws.amazon.com/s3/pricing/ http://aws.amazon.com/s3/pricing/
  - evansd 12y ago
    If you're not in a huge hurry to delete the bucket you can easily set lifecycle rules to expire all versions older than 1 day (the minimum) and then just wait for S3 to delete them for you.

kolev 12y ago

Nice feature, but, again, not available in CloudFormation. I understand the challenge to coordinate releases, I can live with a week or two lag, but usually it's months - CloudFormation still doesn't support basic AWS services. I really am considering abandoning this gem, which is highly neglected and underutilized. Terraform [1] looks like something I can extend myself easily if the upstream doesn't support something just yet and contribute it as a pull request. [1] https://terraform.io/ https://terraform.io/