23 ms·

Use AWS Lambda to self-host the comments for your blog

226 points by jarmitage 10y ago

jimpick 10y ago
Cool. It's great to see my project on Hacker News! Feel free to ask me any questions here, or using the comment system on the blog post. :-)
- ShakataGaNai 10y ago
  Love the idea. For those who aren't as Lambda savvy you might want to provide some "real world" information on pricing. Ex: This code takes roughly 500ms to run, which means you should expect to pay $0.10 per 96,000 comments on your blog. I know it's stupidly small costs (Lambda is wonderful for that), but some people still don't understand how to translate the pricing structure to "the real world".
  - jimpick 10y ago
    Looking at the CloudWatch Logs, each comment seems to take approximately (Billed Duration): 400ms for the initial request, and 800-1000ms for the worker to concatenate the request into the published JSON. It's running in the minimum size of RAM: 128MB
    - prodigal_erik 10y ago
      This is making me wonder whether we should be paying for GC or just having Lambda abandon workers whose heaps are full and spin up more.
      - mrmagooey 10y ago
        I wouldn't have thought the GC would be expensive enough to warrant turning it off, interesting thought though.
      - pmontra 10y ago
        Lambda doesn't destroy the instance at the end of the response. It keeps it running just in case other requests arrive. If that happens you might want GC or a worker eventually runs out of memory. What could happen with Lambda and similar services is that costs will take back under the spotlight "old" languages that run with little memory and little CPU cicles. I'm even thinking about C. I'm not into Go, but that looks like another candidate. Scenarios: those ubiquitous web services running on a single VM because it's more than enough. The customer has to pay for it all the time and could save money if the app can run on Lambda. Enough to pay more in developer time?
        jimpick 10y ago
        What surprised me with Lambda is that it 'freezes' the VM your code is running in, and 'unfreezes' it when more requests come in. Very clever - they can store a lot of frozen VMs on an SSD. It surprised me when I discovered that a cache I had written in my code was already populated! I think most code running on lambda is not going to be CPU bound. In lambda-comments case it's mostly going to be idling as it waits for TCP/IP traffic to/from Akismet, S3, DynamoDB, Slack notification webhooks, etc.
  - cddotdotslash 10y ago
    For anyone looking to calculate Lambda pricing, I made a calculator[1] when developing a previous project. [1] https://s3.amazonaws.com/lambda-tools/pricing-calculator.html https://s3.amazonaws.com/lambda-tools/pricing-calculator.htm...
nickjj 10y ago
Have you been able to quantify the exact pricing per month? From the looks of the git repo it's at least 65 cents per month but then there's a number of other services with no price reference. What would you estimate it would cost per month to host a blog that gets 50 comments per month and what would the cost look like at the end of the year given costs would rise over time due to needing to store more info. For argument's sake, let's assume no AWS free tier.
- jimpick 10y ago
  I don't have "real life" numbers yet, since I just set it up yesterday. I've used up my free tier, so I should be able to get real numbers from Cost Explorer. I'm not sure if everything is tagged nicely for that. On paper, it should cost under a dollar per month. The main cost is just the DynamoDB instance. I don't think it's going to consume much S3 storage. I guess if your blog has a lot of traffic, the network transfer might add up -- but it's just little JSON files.
  - cdnsteve 10y ago
    With the HN burst traffic, it would be good to report back costs after things settle down :)
    - jimpick 10y ago
      Cost report: https://jimpick.com/2016/05/05/introducing-lambda-comments/#comment-BGy5U1MkSzCEAss7bcaVOg https://jimpick.com/2016/05/05/introducing-lambda-comments/#... For 8000 visitors, it cost me $1.37. The numbers might be skewed upwards a bit because somebody posted a gigantic blob for one of the comments to test if I had implemented a limit on the length (I hadn't).
      - nickjj 10y ago
        Thanks for the solid numbers based on real data. At the time of this posting there's 103 comments too which IMO seems waaaaay high given 8k visitors due to the nature of the post. Hard to say for sure but I wouldn't be surprised if a normal blog received 5-6 comments per 10,000 views.
thenomad 10y ago
This is really cool. Nice one. It will desperately need a moderation queue. Even smaller established websites get a ton of comment spam these days. I'd not even consider setting up any commenting system that doesn't have robust spam protection.
- jimpick 10y ago
  That's definitely on the To Do list. I need to add a little authentication interface for the admin - the complexity starts to pile up. But I think it can be done nicely.
  - tracker1 10y ago
    The admin interface could be something that runs local though, would save the time in terms of adding authentication.
    - jimpick 10y ago
      I've already had to edit the JSON file on S3 to remove/edit a couple of comments. That was pretty easy to do from the command-line, but I think most people will want a web interface.
  - Torgo 10y ago
    I have a little bit of code similar to yours for comments, I just wrote some glue to require Google or Facebook authentication through AWS Cognito to make comments. Some people will hate that, but it worked.
- dlhavema 10y ago
  is that something amazon mecahnical turk could be used for?
stephenr 10y ago
A tool that is utterly dependent on 5 aws-specific services is not what I'd call "self hosted". It's also not "serverless" as the repo readme claims.
- jimpick 10y ago
  "Serverless" is the trendy term that the community is describing this particular architecture. I didn't invent the term. http://serverlessconf.io/ http://serverlessconf.io/ https://serverlesscode.com/ https://serverlesscode.com/ https://github.com/serverless/serverless https://github.com/serverless/serverless Of course there are servers out in the cloud somewhere. But you're not renting them 24/7. I'd love to experiment with a completely server-free P2P system, perhaps using WebRTC and WebTorrent (although in that case, there are still some servers involved).
  - stephenr 10y ago
    "Serverless" is the trendy term that the community is describing this particular architecture Describing it extremely poorly, quite frankly. Also - this project is not just consuming resources from the totally not server based resources it inherently relies on, a key component of it runs on those totally not servers. After reading a little on the pages you linked to, "Server-less" is just a 'cute' way to say "our app is vendor-locked to AWS and their Lambda/API gateway/hosted DB/hosted storage/etc services" > Of course there are servers out in the cloud somewhere. Trying to explain a trendy mis-used term with another trendy mis-used term doest't help your case. Have you ever heard the phrase "there is no cloud, it's just someone else's computer"? > But you're not renting them 24/7. So, any application where the server-side component is scaled up-and-down dynamically, based on some kind of workload based schedule, is server-less?
    - dgemm 10y ago
      Yes, "serverless" is a dumb term because we all know there is a server somewhere. ...who cares?
      - 10y ago
dtnewman 10y ago
This is a great idea. I setup something similar for my brother's website contact form (host the site on S3 and then use lambda to shoot him an email when someone submits their data on the contact form). The costs for running his website are just a few cents a month (above the $0.50/month charge for hosting on Route 53). For those who work in Python, there is a project called Zappa (https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa) that allows you to take an existing project in a framework such as Flask and "host" the whole thing with lambda. The project is still young, but I imagine this sort of thing could be very useful to people doing a hobby project that don't want to pay for full-time hosting, but want their site to be available full-time. Hopefully, we get to a point soon where this sort of framework is ready for production use, though I don't know that it's there yet.
- cdnsteve 10y ago
  Wow Zappa looks great, thanks for sharing.
- Aaronontheweb 10y ago
  Not to be an empty-calorie comment, but genuinely thanks for sharing that. Found it to be very helpful!
  - dtnewman 10y ago
    Glad you found it useful. The great thing about Lambda, I think, is the cost, which is so cheap as to be effectively free for a lot of uses (according to https://aws.amazon.com/lambda/pricing/ https://aws.amazon.com/lambda/pricing/, the first million requests each month are free). So if you could design a web service to be deployable on lambda (via zappa or whatever framework), you can effectively host a website for close to no cost assuming you don't have a lot of traffic. On the other hand, you lose some of the control that comes with having your service run on a EC2 instance. For example, I like being able to SSH in to the server to troubleshoot things when they go wrong. But for a hobby project that I put together in a few hours, I might not care enough to pay for the EC2 instance.
    - 10y ago
rando289 10y ago
Use a hosted service to self-host? Come on. Does it also have more cloud in it?
- jimpick 10y ago
  In the blog post, I do link to some projects you can run on your own server. Actually, it wouldn't take much work to modify the code in this project to run in a Node.js daemon. I might just do it anyways to make testing/dev easier.
  - tracker1 10y ago
    Been working, slowly-but surely in terms of writing my own blog engine (static generator) and then publishing to gh-pages, but was considering using discus for comments... thinking of doing something similar as a docker contained service that I can throw up on my dokku server. TBH, the thought of managing the attack surface of comments is kind of scary... Will definitely be referring back to this.
    - jimpick 10y ago
      My approach was to just pump the entire comment through the "markdown-it" markdown processor, which promises to emit safe HTML. It's a popular project, so I'm banking on the fact that they go a good job of sanitizing in their pipeline.
- startling 10y ago
  Using hardware you didn't design to self-host? Come on.
euphemize 10y ago
I am not sure when this was written, but Lambda supports ES6 directly as of about a month ago. https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-now-available-on-lambda/ https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-n... I think you can avoid the transpiling/babel dependency. Great work!
- jimpick 10y ago
  Yep, I started it before that. I'd love to lessen the dependency on Babel/Webpack as much as possible. I'm still using async/await, ES6 modules and some other more experiment ES-next features, so I think it will be awhile before Babel/Webpack can be completely removed. Of course, I don't really have to use those features. :-)
  - tracker1 10y ago
    If the babel/webpack builds happen before deployment to Lambda, I'm not sure it really matters at this point... once we see Node > 7 or so drop, with ES6 module support, it might then be time to consider dropping babel, but it's going to be around a while, and still nicer in a lot of ways than alternatives.
eistrati 10y ago
Great work! The fact that data is stored in my account (to be read: under my control) instead of outsourcing to 3rd parties is a huge plus. If you will be able to make it less "technical" and more "plug-and-play", I think this will be cool viable solution to integrate (to be read: for customers like ours, that uses serverless computing in enterprise apps -> https://github.com/MitocGroup/deep-framework https://github.com/MitocGroup/deep-framework).
seandavidfisher 10y ago
One step closer to my dream of "Wordpress on Lambda"! What other projects exist to run a blog on Lambda? Apart from going full in on a locally-compiled static site that I just upload to S3, that is? (Hugo, Hexo, Jekyll, etc) I'm imagining a full admin web interface that uses a Lambda-driven API, themes, a front page with dynamically generated content from the Lambda-driven API, etc.
- xemoka 10y ago
  Zappa is pretty cool, I haven't had a chance to use it myself but want to find a project to apply django-zappa ( https://github.com/Miserlou/django-zappa https://github.com/Miserlou/django-zappa ) on django-cms or wagtail... not too sure how that'd work out though.
  - Mizza 10y ago
    Zappa author here - Wagtail works great on Django-Zappa. I have also been slowly writing my own CMS, https://github.com/miserlou/zappa-cms https://github.com/miserlou/zappa-cms, although that's not ready for public consumption yet!
hn_user2 10y ago
This is really cool to see! I have had this type of thing on a back burner for a long time. Glad to see someone wrote it. I wonder if the author considered AWS SimpleDB instead of dynamo. Costs are a bit cheaper. https://aws.amazon.com/simpledb/ https://aws.amazon.com/simpledb/
- wsh91 10y ago
  SimpleDB hasn't been touched for a long time. It seems to be deprecated.
- jimpick 10y ago
  I'm not really using DynamoDB to store data. I'm using it to sequence parallel requests going into the 'worker' that concatenates things together -- so two simultaneous incoming comments don't clobber each other. At the moment, I think only DynamoDB Streams and Kinesis event sources will batch events together and deliver them to a single function (instead of many parallel functions). It's hard to understand just by reading the docs. http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html http://docs.aws.amazon.com/lambda/latest/dg/intro-core-compo... I think the platform is evolving quickly, so I wouldn't be surprised if they come up with some more options in the future. I'm using DynamoDB streams since it's cheaper than Kinesis.
  - Arelius 10y ago
    The problem with DynamoDB is it has minimum provisioned throughput requirements that would make this rather expensive for a blog that was not free-tier eligible even if it got no traffic at all. In contrast, SimpleDB seems to only charge for actual throughput used.
    - jimpick 10y ago
      Correct. I'm provisioning the minimum amount. I'd love to find a cheaper solution.
  - hrez 10y ago
    Why not SQS?
akkartik 10y ago
Very interesting! I'd like to migrate off Disqus, but I'd like to not leave my old comments orphaned. Do people know of any export services? (This may be an ill-posed question since comments arguably belong to their authors..)
alasano 10y ago
I just finished setting up my personal site with Hugo+S3+Lambda yesterday. Lambda runs Hugo when the S3 bucket detects a file change, takes about 5000 ms per blog post which is fractions of a penny. Although this adds a bit of complexity compared to Disqus it sounds like a fun solution to implement, potentially with the addition of a way to moderate comments.
- jimpick 10y ago
  A future project for me is to see if I can render the comments out to a string at the moment when Hugo is rebuilding my blog. Then the comments would be in the DOM before the JavaScript is even loaded. I've done that with React apps before - it works great. If that was working, I could rebuild the blog after every comment even, using a system like yours.
  - alasano 10y ago
    I think the serverless(ish) space is headed in an exciting direction overall, especially if we can drastically reduce wasted compute time. I'm watching your repo and hopefully will get inspired! Thanks for the reply Jim
- biokoda 10y ago
  I hate disqus for having no search option. You see traffic coming from them and there is no way to figure out where those comments are. Any information written in disqus comments is only slightly more accessible than the dark web.
  - dexterdog 10y ago
    Plus the whole tracking issue. I assume most people with privacy blockers are blocking disqus anyway.
- james-skemp 10y ago
  Did you follow a tutorial by chance? Moved one of my sites over to GitHub and Hugo recently and the ease of Hugo has made me seriously consider moving as much as I can of my other stuff.
wnevets 10y ago
I used google sheets as my database for a similar setup with s3 and lambda. Sheets is completely is free AFAIK
- imaginenore 10y ago
  Me too. And it has an API.
amelius 10y ago
I wish more people added a comments section to the documentation of their software projects.
- jimpick 10y ago
  I think it might be possible to embed lambda-comments into an iframe that could then appear on GitHub README.md files, etc. It's just an untested theory for now. ;-)
  - kasbah 10y ago
    I don't think you can. http://stackoverflow.com/questions/11804820/embed-a-you-tube-video http://stackoverflow.com/questions/11804820/embed-a-you-tube... You might be able to on Wikis though.
cowholio4 10y ago
We use Lambda for our contact form; it's great. Our website is completely hosted on s3/cloudfront so it's much cheaper and stable than running a dedicated machine. If I get some time this weekend, I'll clean it and push it up to github. [EDIT] Here it is. This has been working for us for almost a year. If there is interest, I can clean it up. https://github.com/cowholio4/lambda-contact-form https://github.com/cowholio4/lambda-contact-form
- wallacoloo 10y ago
  Oh, this is great - thanks! I've been considering adding comments to my static website for a while. I already deploy the website over IPFS, so that [in theory] the entire website can be distributed, and it can be duplicated/cached, etc when I have a spotty internet connection. Ideally, I would also have the comments published directly on IPFS as well, rather than having the comments be retrieved from somewhere external to the IPFS network. Your code should let me deal with that - just have the comment system fire off an email to me when a user posts a comment, and a daemon running on my PC can publish the comment next time it's online. Plus, I get easy email notifications :) Such an approach seems a bit convoluted though. I'm not used to web programming, wherein it seems like every popular approach to hosting something as simple as a blog comes with pretty significant drawbacks or limitations. I'm much more fond of working in environments which offer an objectively "right" way to do every task.
- appleflaxen 10y ago
  How does it work? You POST to the lambda service, and it emails the url parameters with the specified formatting? This requires a separate HTML form to be served from your website first, right? Very cool idea! Thanks for sharing it :)
  - cowholio4 10y ago
    Yeah, I didn't really make that part clear. You have to create an API Gateway. Luckily, AWS lets you share API definitions so I added that to the repo. It's the swagger.json file. I'll try to get this all automated to make it super simple, but for now you just have to replace all those placeholders.
pjc50 10y ago
I appreciate your realisation that spam and abuse are #1 implementation problems in commenting systems.
mattbeckman 10y ago
Lambda is great. Few friends and I just put up what we call LDAN (Lambda, DynamoDB, Aurelia, and Node) aka Lieutenant Dan. Pretty cool refreshing change that is in stark contrast to our larger EC2+MongoDB stack. http://ldan.io/ http://ldan.io/
- pmontra 10y ago
  I went for the demo and I stopped right away when it asked me to login with Google or Facebook. I went to the repo and it's very light on documentation. At least I learnt about aurelia.io
jrockway 10y ago
I eliminated all spam and abuse from my blog by requiring comments to be PGP-signed :) I did still get some comments too!
- jimpick 10y ago
  That's pretty extreme. I like it.
- copperx 10y ago
  The statement in your website "On some level, the era of personal websites is over" made me sad and nostalgic. I'll keep it old-school by maintaining my own personal website.
  - imglorp 10y ago
    I don't think that's true. It might at least be accurate to say there's an end to open comment sections and other abuse magnets. Anybody can still make a site in a few minutes on a number of places, they just aren't going to solve the abuse problem themselves without using a hardened tool.
nzoschke 10y ago
This is extremely cool. I'm starting to see a serious new trend, where automating around IaaS can completely eliminate the need for SaaS. Why pay for a subscription comment service when it can be implemented as a script in the cloud that only runs on-demand?
jimpick 10y ago
Here's a little timelapse video of the Google Analytics real-time page showing 4 hours of being on the home page of Hacker News. :-) https://twitter.com/jimpick/status/728712464687239168 https://twitter.com/jimpick/status/728712464687239168
yarper 10y ago
You could also use elastic-beanstalk, which I think is also free for very low traffic sites.
- jimpick 10y ago
  AWS has a free tier that gives you some free resources for the first 12 months. After that, you have to pay.
jasonszhao 10y ago
Funny how it says at the bottom: "Error loading comments."
- jimpick 10y ago
  I haven't done lots of testing on it yet. I just found out that happens when browsing in 'Incognito' mode. Probably a bug.
  - jimpick 10y ago
    I was using localStorage, which apparently you can't do if you are on Safari running in private mode. I've changed the code, and it should work now.
e12e 10y ago
Nice project. I wish there was away to get stuff like this to work in a way that didn't take an all-or-nothing approach to noscript-filtering. As it is, now I have the option of no comments, or allowing scripts from: "s3-us-west-2.amazonaws.com", which almost might as well be "anywhere and everywhere". FWIW I see a similar problem with cloudfront and CDNs in general.
pitchka 10y ago
Why not just adapt Isso for this? It already has a bunch of nice functionality. I personally just delegate the commenting to github and render the comments with javascript through github api. http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-github-hosted-blogs.html http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-... Here's a nice overview of how this works.
- jimpick 10y ago
  That's pretty cool.
bikamonki 10y ago
As you describe the goals and features of your system, I think the AWS solution is too complicated, even over-kill, for what you are trying to achive. I use AWS extensively for the 'big projects', I also use it as a CDN for all projects, but for the small projects (in terms of complexity and capacity), I find it much simpler to implement a two-tier service. For example, for comments (form submission in general), it is much simpler to write a node app, host it on a free-tier and persist the data also on a free-tier (for example Heroku+Firebase). Parse.com was great b/c you could do it all in one tier since the offered server side logic. Granted, if you ever pass the free tier limits, normally the price breaks of these PAAS providers are high so cost does not scale 'by the cent' like AWS; however, if you pass the free tier you must be making good traffic which should translate to good income so infra cost shouldn't be a problem.
- bikamonki 10y ago
  Downvote all you want but IMO this is a Rube Goldberg machine that uses 5!!! AWS services to persist what? 5k comments for the entire lifecycle of the website? The modern programmer is quite good at playing Legos with the myriad of micro webservices out there (which is good play/training) but has forgotten how to think like an 'engineer' must think.
  - jimpick 10y ago
    Any working solution actually has thousands of components. I'd defend this particular Rube Goldberg machine by pointing out that 99% of the time, none of my code is running on the server side - it's all Amazon S3. The only time my code runs in the cloud is when a comment gets submitted. I'd argue that it's pretty easy to reason about. I've got some more blog posts coming. :-)
    - bikamonki 10y ago
      Don't get me wrong, what you've done is how we will build web stack solutions from now on: orchestrated microservices. My hopefully constructive critique is that as engineers we ought to find the best solution for a given problem, even at the expense of such solution not being universal. Take the myriad of tiny to medium websites that use WP, it is like renting an 18-wheeler to move one tiny box. The engineered correct solution would be a static site. Now, for a site with just a handful of posts and comments we should think of a much simpler solution for comments persistence. In fact, after reading about your software I am inspired to think and try to come up with such solution, let's see :)
ekiara 10y ago
Pricing for AWS Lambda is complicated, but reading the examples clarifies things... a bit. Is The Lambda free tier something separate from the free tier you get on first signing up for AWS? I used up my AWS EC2 free tier years ago and I don't think you are eligible again once you use your free tier. I wish they'd put regular pricing at the top instead, i.e. for someone who doesn't qualify for the free tier you start paying from request #1.

226 points by jarmitage 10y ago

Cool. It's great to see my project on Hacker News! Feel free to ask me any questions here, or using the comment system on the blog post. :-)

ShakataGaNai 10y ago
Love the idea. For those who aren't as Lambda savvy you might want to provide some "real world" information on pricing. Ex: This code takes roughly 500ms to run, which means you should expect to pay $0.10 per 96,000 comments on your blog. I know it's stupidly small costs (Lambda is wonderful for that), but some people still don't understand how to translate the pricing structure to "the real world".
- jimpick 10y ago
  Looking at the CloudWatch Logs, each comment seems to take approximately (Billed Duration): 400ms for the initial request, and 800-1000ms for the worker to concatenate the request into the published JSON. It's running in the minimum size of RAM: 128MB
  - prodigal_erik 10y ago
    This is making me wonder whether we should be paying for GC or just having Lambda abandon workers whose heaps are full and spin up more.
    - mrmagooey 10y ago
      I wouldn't have thought the GC would be expensive enough to warrant turning it off, interesting thought though.
    - pmontra 10y ago
      Lambda doesn't destroy the instance at the end of the response. It keeps it running just in case other requests arrive. If that happens you might want GC or a worker eventually runs out of memory. What could happen with Lambda and similar services is that costs will take back under the spotlight "old" languages that run with little memory and little CPU cicles. I'm even thinking about C. I'm not into Go, but that looks like another candidate. Scenarios: those ubiquitous web services running on a single VM because it's more than enough. The customer has to pay for it all the time and could save money if the app can run on Lambda. Enough to pay more in developer time?
      - jimpick 10y ago
        What surprised me with Lambda is that it 'freezes' the VM your code is running in, and 'unfreezes' it when more requests come in. Very clever - they can store a lot of frozen VMs on an SSD. It surprised me when I discovered that a cache I had written in my code was already populated! I think most code running on lambda is not going to be CPU bound. In lambda-comments case it's mostly going to be idling as it waits for TCP/IP traffic to/from Akismet, S3, DynamoDB, Slack notification webhooks, etc.
- cddotdotslash 10y ago
  For anyone looking to calculate Lambda pricing, I made a calculator[1] when developing a previous project. [1] https://s3.amazonaws.com/lambda-tools/pricing-calculator.html https://s3.amazonaws.com/lambda-tools/pricing-calculator.htm...

nickjj 10y ago

Have you been able to quantify the exact pricing per month? From the looks of the git repo it's at least 65 cents per month but then there's a number of other services with no price reference. What would you estimate it would cost per month to host a blog that gets 50 comments per month and what would the cost look like at the end of the year given costs would rise over time due to needing to store more info. For argument's sake, let's assume no AWS free tier.

jimpick 10y ago
I don't have "real life" numbers yet, since I just set it up yesterday. I've used up my free tier, so I should be able to get real numbers from Cost Explorer. I'm not sure if everything is tagged nicely for that. On paper, it should cost under a dollar per month. The main cost is just the DynamoDB instance. I don't think it's going to consume much S3 storage. I guess if your blog has a lot of traffic, the network transfer might add up -- but it's just little JSON files.
- cdnsteve 10y ago
  With the HN burst traffic, it would be good to report back costs after things settle down :)
  - jimpick 10y ago
    Cost report: https://jimpick.com/2016/05/05/introducing-lambda-comments/#comment-BGy5U1MkSzCEAss7bcaVOg https://jimpick.com/2016/05/05/introducing-lambda-comments/#... For 8000 visitors, it cost me $1.37. The numbers might be skewed upwards a bit because somebody posted a gigantic blob for one of the comments to test if I had implemented a limit on the length (I hadn't).
    - nickjj 10y ago
      Thanks for the solid numbers based on real data. At the time of this posting there's 103 comments too which IMO seems waaaaay high given 8k visitors due to the nature of the post. Hard to say for sure but I wouldn't be surprised if a normal blog received 5-6 comments per 10,000 views.

thenomad 10y ago

This is really cool. Nice one. It will desperately need a moderation queue. Even smaller established websites get a ton of comment spam these days. I'd not even consider setting up any commenting system that doesn't have robust spam protection.

jimpick 10y ago
That's definitely on the To Do list. I need to add a little authentication interface for the admin - the complexity starts to pile up. But I think it can be done nicely.
- tracker1 10y ago
  The admin interface could be something that runs local though, would save the time in terms of adding authentication.
  - jimpick 10y ago
    I've already had to edit the JSON file on S3 to remove/edit a couple of comments. That was pretty easy to do from the command-line, but I think most people will want a web interface.
- Torgo 10y ago
  I have a little bit of code similar to yours for comments, I just wrote some glue to require Google or Facebook authentication through AWS Cognito to make comments. Some people will hate that, but it worked.
dlhavema 10y ago
is that something amazon mecahnical turk could be used for?

stephenr 10y ago

A tool that is utterly dependent on 5 aws-specific services is not what I'd call "self hosted". It's also not "serverless" as the repo readme claims.

jimpick 10y ago
"Serverless" is the trendy term that the community is describing this particular architecture. I didn't invent the term. http://serverlessconf.io/ http://serverlessconf.io/ https://serverlesscode.com/ https://serverlesscode.com/ https://github.com/serverless/serverless https://github.com/serverless/serverless Of course there are servers out in the cloud somewhere. But you're not renting them 24/7. I'd love to experiment with a completely server-free P2P system, perhaps using WebRTC and WebTorrent (although in that case, there are still some servers involved).
- stephenr 10y ago
  "Serverless" is the trendy term that the community is describing this particular architecture Describing it extremely poorly, quite frankly. Also - this project is not just consuming resources from the totally not server based resources it inherently relies on, a key component of it runs on those totally not servers. After reading a little on the pages you linked to, "Server-less" is just a 'cute' way to say "our app is vendor-locked to AWS and their Lambda/API gateway/hosted DB/hosted storage/etc services" > Of course there are servers out in the cloud somewhere. Trying to explain a trendy mis-used term with another trendy mis-used term doest't help your case. Have you ever heard the phrase "there is no cloud, it's just someone else's computer"? > But you're not renting them 24/7. So, any application where the server-side component is scaled up-and-down dynamically, based on some kind of workload based schedule, is server-less?
  - dgemm 10y ago
    Yes, "serverless" is a dumb term because we all know there is a server somewhere. ...who cares?
    - 10y ago

dtnewman 10y ago

This is a great idea. I setup something similar for my brother's website contact form (host the site on S3 and then use lambda to shoot him an email when someone submits their data on the contact form). The costs for running his website are just a few cents a month (above the $0.50/month charge for hosting on Route 53). For those who work in Python, there is a project called Zappa (https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa) that allows you to take an existing project in a framework such as Flask and "host" the whole thing with lambda. The project is still young, but I imagine this sort of thing could be very useful to people doing a hobby project that don't want to pay for full-time hosting, but want their site to be available full-time. Hopefully, we get to a point soon where this sort of framework is ready for production use, though I don't know that it's there yet.

cdnsteve 10y ago
Wow Zappa looks great, thanks for sharing.
Aaronontheweb 10y ago
Not to be an empty-calorie comment, but genuinely thanks for sharing that. Found it to be very helpful!
- dtnewman 10y ago
  Glad you found it useful. The great thing about Lambda, I think, is the cost, which is so cheap as to be effectively free for a lot of uses (according to https://aws.amazon.com/lambda/pricing/ https://aws.amazon.com/lambda/pricing/, the first million requests each month are free). So if you could design a web service to be deployable on lambda (via zappa or whatever framework), you can effectively host a website for close to no cost assuming you don't have a lot of traffic. On the other hand, you lose some of the control that comes with having your service run on a EC2 instance. For example, I like being able to SSH in to the server to troubleshoot things when they go wrong. But for a hobby project that I put together in a few hours, I might not care enough to pay for the EC2 instance.
  - 10y ago

rando289 10y ago

Use a hosted service to self-host? Come on. Does it also have more cloud in it?

jimpick 10y ago
In the blog post, I do link to some projects you can run on your own server. Actually, it wouldn't take much work to modify the code in this project to run in a Node.js daemon. I might just do it anyways to make testing/dev easier.
- tracker1 10y ago
  Been working, slowly-but surely in terms of writing my own blog engine (static generator) and then publishing to gh-pages, but was considering using discus for comments... thinking of doing something similar as a docker contained service that I can throw up on my dokku server. TBH, the thought of managing the attack surface of comments is kind of scary... Will definitely be referring back to this.
  - jimpick 10y ago
    My approach was to just pump the entire comment through the "markdown-it" markdown processor, which promises to emit safe HTML. It's a popular project, so I'm banking on the fact that they go a good job of sanitizing in their pipeline.
startling 10y ago
Using hardware you didn't design to self-host? Come on.

euphemize 10y ago

I am not sure when this was written, but Lambda supports ES6 directly as of about a month ago. https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-now-available-on-lambda/ https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-n... I think you can avoid the transpiling/babel dependency. Great work!

jimpick 10y ago
Yep, I started it before that. I'd love to lessen the dependency on Babel/Webpack as much as possible. I'm still using async/await, ES6 modules and some other more experiment ES-next features, so I think it will be awhile before Babel/Webpack can be completely removed. Of course, I don't really have to use those features. :-)
- tracker1 10y ago
  If the babel/webpack builds happen before deployment to Lambda, I'm not sure it really matters at this point... once we see Node > 7 or so drop, with ES6 module support, it might then be time to consider dropping babel, but it's going to be around a while, and still nicer in a lot of ways than alternatives.

eistrati 10y ago

Great work! The fact that data is stored in my account (to be read: under my control) instead of outsourcing to 3rd parties is a huge plus. If you will be able to make it less "technical" and more "plug-and-play", I think this will be cool viable solution to integrate (to be read: for customers like ours, that uses serverless computing in enterprise apps -> https://github.com/MitocGroup/deep-framework https://github.com/MitocGroup/deep-framework).

seandavidfisher 10y ago

One step closer to my dream of "Wordpress on Lambda"! What other projects exist to run a blog on Lambda? Apart from going full in on a locally-compiled static site that I just upload to S3, that is? (Hugo, Hexo, Jekyll, etc) I'm imagining a full admin web interface that uses a Lambda-driven API, themes, a front page with dynamically generated content from the Lambda-driven API, etc.

xemoka 10y ago
Zappa is pretty cool, I haven't had a chance to use it myself but want to find a project to apply django-zappa ( https://github.com/Miserlou/django-zappa https://github.com/Miserlou/django-zappa ) on django-cms or wagtail... not too sure how that'd work out though.
- Mizza 10y ago
  Zappa author here - Wagtail works great on Django-Zappa. I have also been slowly writing my own CMS, https://github.com/miserlou/zappa-cms https://github.com/miserlou/zappa-cms, although that's not ready for public consumption yet!

hn_user2 10y ago

This is really cool to see! I have had this type of thing on a back burner for a long time. Glad to see someone wrote it. I wonder if the author considered AWS SimpleDB instead of dynamo. Costs are a bit cheaper. https://aws.amazon.com/simpledb/ https://aws.amazon.com/simpledb/

wsh91 10y ago
SimpleDB hasn't been touched for a long time. It seems to be deprecated.
jimpick 10y ago
I'm not really using DynamoDB to store data. I'm using it to sequence parallel requests going into the 'worker' that concatenates things together -- so two simultaneous incoming comments don't clobber each other. At the moment, I think only DynamoDB Streams and Kinesis event sources will batch events together and deliver them to a single function (instead of many parallel functions). It's hard to understand just by reading the docs. http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html http://docs.aws.amazon.com/lambda/latest/dg/intro-core-compo... I think the platform is evolving quickly, so I wouldn't be surprised if they come up with some more options in the future. I'm using DynamoDB streams since it's cheaper than Kinesis.
- Arelius 10y ago
  The problem with DynamoDB is it has minimum provisioned throughput requirements that would make this rather expensive for a blog that was not free-tier eligible even if it got no traffic at all. In contrast, SimpleDB seems to only charge for actual throughput used.
  - jimpick 10y ago
    Correct. I'm provisioning the minimum amount. I'd love to find a cheaper solution.
- hrez 10y ago
  Why not SQS?

akkartik 10y ago

Very interesting! I'd like to migrate off Disqus, but I'd like to not leave my old comments orphaned. Do people know of any export services? (This may be an ill-posed question since comments arguably belong to their authors..)

alasano 10y ago

I just finished setting up my personal site with Hugo+S3+Lambda yesterday. Lambda runs Hugo when the S3 bucket detects a file change, takes about 5000 ms per blog post which is fractions of a penny. Although this adds a bit of complexity compared to Disqus it sounds like a fun solution to implement, potentially with the addition of a way to moderate comments.

jimpick 10y ago
A future project for me is to see if I can render the comments out to a string at the moment when Hugo is rebuilding my blog. Then the comments would be in the DOM before the JavaScript is even loaded. I've done that with React apps before - it works great. If that was working, I could rebuild the blog after every comment even, using a system like yours.
- alasano 10y ago
  I think the serverless(ish) space is headed in an exciting direction overall, especially if we can drastically reduce wasted compute time. I'm watching your repo and hopefully will get inspired! Thanks for the reply Jim
biokoda 10y ago
I hate disqus for having no search option. You see traffic coming from them and there is no way to figure out where those comments are. Any information written in disqus comments is only slightly more accessible than the dark web.
- dexterdog 10y ago
  Plus the whole tracking issue. I assume most people with privacy blockers are blocking disqus anyway.
james-skemp 10y ago
Did you follow a tutorial by chance? Moved one of my sites over to GitHub and Hugo recently and the ease of Hugo has made me seriously consider moving as much as I can of my other stuff.

wnevets 10y ago

I used google sheets as my database for a similar setup with s3 and lambda. Sheets is completely is free AFAIK

imaginenore 10y ago
Me too. And it has an API.

amelius 10y ago

I wish more people added a comments section to the documentation of their software projects.

jimpick 10y ago
I think it might be possible to embed lambda-comments into an iframe that could then appear on GitHub README.md files, etc. It's just an untested theory for now. ;-)
- kasbah 10y ago
  I don't think you can. http://stackoverflow.com/questions/11804820/embed-a-you-tube-video http://stackoverflow.com/questions/11804820/embed-a-you-tube... You might be able to on Wikis though.

cowholio4 10y ago

We use Lambda for our contact form; it's great. Our website is completely hosted on s3/cloudfront so it's much cheaper and stable than running a dedicated machine. If I get some time this weekend, I'll clean it and push it up to github. [EDIT] Here it is. This has been working for us for almost a year. If there is interest, I can clean it up. https://github.com/cowholio4/lambda-contact-form https://github.com/cowholio4/lambda-contact-form

wallacoloo 10y ago
Oh, this is great - thanks! I've been considering adding comments to my static website for a while. I already deploy the website over IPFS, so that [in theory] the entire website can be distributed, and it can be duplicated/cached, etc when I have a spotty internet connection. Ideally, I would also have the comments published directly on IPFS as well, rather than having the comments be retrieved from somewhere external to the IPFS network. Your code should let me deal with that - just have the comment system fire off an email to me when a user posts a comment, and a daemon running on my PC can publish the comment next time it's online. Plus, I get easy email notifications :) Such an approach seems a bit convoluted though. I'm not used to web programming, wherein it seems like every popular approach to hosting something as simple as a blog comes with pretty significant drawbacks or limitations. I'm much more fond of working in environments which offer an objectively "right" way to do every task.
appleflaxen 10y ago
How does it work? You POST to the lambda service, and it emails the url parameters with the specified formatting? This requires a separate HTML form to be served from your website first, right? Very cool idea! Thanks for sharing it :)
- cowholio4 10y ago
  Yeah, I didn't really make that part clear. You have to create an API Gateway. Luckily, AWS lets you share API definitions so I added that to the repo. It's the swagger.json file. I'll try to get this all automated to make it super simple, but for now you just have to replace all those placeholders.

pjc50 10y ago

I appreciate your realisation that spam and abuse are #1 implementation problems in commenting systems.

mattbeckman 10y ago

Lambda is great. Few friends and I just put up what we call LDAN (Lambda, DynamoDB, Aurelia, and Node) aka Lieutenant Dan. Pretty cool refreshing change that is in stark contrast to our larger EC2+MongoDB stack. http://ldan.io/ http://ldan.io/

pmontra 10y ago
I went for the demo and I stopped right away when it asked me to login with Google or Facebook. I went to the repo and it's very light on documentation. At least I learnt about aurelia.io

jrockway 10y ago

I eliminated all spam and abuse from my blog by requiring comments to be PGP-signed :) I did still get some comments too!

jimpick 10y ago
That's pretty extreme. I like it.
copperx 10y ago
The statement in your website "On some level, the era of personal websites is over" made me sad and nostalgic. I'll keep it old-school by maintaining my own personal website.
- imglorp 10y ago
  I don't think that's true. It might at least be accurate to say there's an end to open comment sections and other abuse magnets. Anybody can still make a site in a few minutes on a number of places, they just aren't going to solve the abuse problem themselves without using a hardened tool.

nzoschke 10y ago

This is extremely cool. I'm starting to see a serious new trend, where automating around IaaS can completely eliminate the need for SaaS. Why pay for a subscription comment service when it can be implemented as a script in the cloud that only runs on-demand?

jimpick 10y ago

Here's a little timelapse video of the Google Analytics real-time page showing 4 hours of being on the home page of Hacker News. :-) https://twitter.com/jimpick/status/728712464687239168 https://twitter.com/jimpick/status/728712464687239168

yarper 10y ago

You could also use elastic-beanstalk, which I think is also free for very low traffic sites.

jimpick 10y ago
AWS has a free tier that gives you some free resources for the first 12 months. After that, you have to pay.

jasonszhao 10y ago

Funny how it says at the bottom: "Error loading comments."

jimpick 10y ago
I haven't done lots of testing on it yet. I just found out that happens when browsing in 'Incognito' mode. Probably a bug.
- jimpick 10y ago
  I was using localStorage, which apparently you can't do if you are on Safari running in private mode. I've changed the code, and it should work now.

e12e 10y ago

Nice project. I wish there was away to get stuff like this to work in a way that didn't take an all-or-nothing approach to noscript-filtering. As it is, now I have the option of no comments, or allowing scripts from: "s3-us-west-2.amazonaws.com", which almost might as well be "anywhere and everywhere". FWIW I see a similar problem with cloudfront and CDNs in general.

pitchka 10y ago

Why not just adapt Isso for this? It already has a bunch of nice functionality. I personally just delegate the commenting to github and render the comments with javascript through github api. http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-github-hosted-blogs.html http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-... Here's a nice overview of how this works.

jimpick 10y ago
That's pretty cool.

bikamonki 10y ago

As you describe the goals and features of your system, I think the AWS solution is too complicated, even over-kill, for what you are trying to achive. I use AWS extensively for the 'big projects', I also use it as a CDN for all projects, but for the small projects (in terms of complexity and capacity), I find it much simpler to implement a two-tier service. For example, for comments (form submission in general), it is much simpler to write a node app, host it on a free-tier and persist the data also on a free-tier (for example Heroku+Firebase). Parse.com was great b/c you could do it all in one tier since the offered server side logic. Granted, if you ever pass the free tier limits, normally the price breaks of these PAAS providers are high so cost does not scale 'by the cent' like AWS; however, if you pass the free tier you must be making good traffic which should translate to good income so infra cost shouldn't be a problem.

bikamonki 10y ago
Downvote all you want but IMO this is a Rube Goldberg machine that uses 5!!! AWS services to persist what? 5k comments for the entire lifecycle of the website? The modern programmer is quite good at playing Legos with the myriad of micro webservices out there (which is good play/training) but has forgotten how to think like an 'engineer' must think.
- jimpick 10y ago
  Any working solution actually has thousands of components. I'd defend this particular Rube Goldberg machine by pointing out that 99% of the time, none of my code is running on the server side - it's all Amazon S3. The only time my code runs in the cloud is when a comment gets submitted. I'd argue that it's pretty easy to reason about. I've got some more blog posts coming. :-)
  - bikamonki 10y ago
    Don't get me wrong, what you've done is how we will build web stack solutions from now on: orchestrated microservices. My hopefully constructive critique is that as engineers we ought to find the best solution for a given problem, even at the expense of such solution not being universal. Take the myriad of tiny to medium websites that use WP, it is like renting an 18-wheeler to move one tiny box. The engineered correct solution would be a static site. Now, for a site with just a handful of posts and comments we should think of a much simpler solution for comments persistence. In fact, after reading about your software I am inspired to think and try to come up with such solution, let's see :)

ekiara 10y ago

Pricing for AWS Lambda is complicated, but reading the examples clarifies things... a bit. Is The Lambda free tier something separate from the free tier you get on first signing up for AWS? I used up my AWS EC2 free tier years ago and I don't think you are eligible again once you use your free tier. I wish they'd put regular pricing at the top instead, i.e. for someone who doesn't qualify for the free tier you start paying from request #1.