6 ms·

Static Site Post Scheduler: Using AWS Lambda and Serverless to Schedule Blog Posts

91 points by davidwells 10y ago

fenollp 10y ago
http://explainshell.com/explain?cmd=at http://explainshell.com/explain?cmd=at Or use a CI cron running daily?
- Lord_Zero 10y ago
  Yeah, they said its "serverless" but it requires an AWS lambda function. I use Jekyll for my blog but I simply commit, push, and run a script that copies the files to S3. https://www.tberra.com/aws/amazon/meta/2016/11/12/the-birth-of-a-blog/ https://www.tberra.com/aws/amazon/meta/2016/11/12/the-birth-... its not scheduled but I could easily tell linux to run that command once at whatever time I wanted.
  - g_delgado14 10y ago
    AWS lambda functions __are__ serverless. There's no notion of uptime, provisioning or managing servers when using lambda.
    - Lord_Zero 10y ago
      This is true, but what triggers the lambda function to run? Where does the lambda function run when its triggered?
      - climber604 10y ago
        Clearly you need to learn more about lambda https://aws.amazon.com/lambda/ https://aws.amazon.com/lambda/
      - davidwells 10y ago
        One lambda runs when github webhooks are fired: https://github.com/serverless/post-scheduler#github-webhook-architecture-overview https://github.com/serverless/post-scheduler#github-webhook-... The other lambda fires on a cron job https://github.com/serverless/post-scheduler#cron-job-architecture-overview https://github.com/serverless/post-scheduler#cron-job-archit...
      - derefr 10y ago
        "Serverless" really means "your code is run as a container with a finite lifetime on an arbitrary machine in a standardized, non-configurable PaaS machine cluster." In essence, it's a return to the "cgi-bin" model of scripting, where you write a script that only runs for the lifetime of one request, and then a web server will spin it up to respond to a relevant request and it'll terminate at the end of that request. The only differences in the modern instantiation of the concept, are that: 1. the "cgi-bin" directory is a mounted object store, common to an entire cluster; and 2. there's some fancy preloading logic that lets Lambda functions stay "hot" between calls to them, so that they can run very quickly. (Sort of like Phusion Passenger does for Ruby code, except with enough isolation to make it safe for multitenant environments. Heroku's old Alpine stack might be a better comparison.)
    - chriswarbo 10y ago
      > There's no notion of uptime, provisioning or managing servers when using lambda. There's no notion of uptime, provisioning or managing Web sites when using Web sites; but the operators of the Web site certainly have to tackle such things. Likewise, I'm sure the operators of AWS lambda are very much concerned about such things, even if the users aren't. In that sense, wordpress.com is "serverless".
      - Spivak 10y ago
        This is really just a semantic argument. You're free to call it Function as a Service or FaaS if you want. Hosted WordPress is SaaS but not FaaS.
  - rhizome 10y ago
    or add a cron job to `git mv` a file from `pending` to `current` and run the commit/publish cycle.
deleted 10y ago
[deleted]
crisopolis 10y ago
Since serverless uses AWS primarily. I'm assuming if I put all my eggs in the serverless basket and Amazon has some amazing outage. I'm screwed? To which I don't understand the appeal of serverless besides having no servers lol.
- davidwells 10y ago
  Hello there! The serverless framework supports a number of different FAAS (function as a service) providers like IBM openwhisk, AWS lambda, & Azure functions from microsoft. See https://serverless.com/framework/docs/providers/ https://serverless.com/framework/docs/providers/ for more info. So it's up to you where your code runs =)
- intrasight 10y ago
  And it's still a static web site, so you're only "screwed" in terms of updates. And if you can't wait for Lambda to be restored, either run the script locally or manually hot-fix those static HTML files.
foxhop 10y ago
This solution is sort of weird to me. To me, the allure of a static site or a static blog is to simplify the stack; To bring it back to the basics. Static HTML served by a good ole-fashion web server. Adding the cognitive burden of "serverless" and/or lambda just to schedule seems to complicate something that isn't very complicated. Its almost like an invented problem. That said, long live this static site generator movement. I'm betting this movement will continue to grow which is why I'm bootstrapping https://www.remarkbox.com https://www.remarkbox.com (comments-as-a-service)
- cocktailpeanuts 10y ago
  I use jekyll NOT because it's simple (Actually it's 100 times more complex to use than a simple wordpress or medium) but because: 1. I can keep track changes with git 2. I can host them on Github pages for FREE 3. I own the content 4. More flexible In fact I've been really annoyed with all the boilerplate code and extensions I need to deal with in order to have a "usable" workflow set up, I almost thought about moving back to free blog services like medium, tumblr, wordpress, etc. but the only reasons I can't are the ones I mentioned above.
  - foxhop 10y ago
    Jekyll is way more simple from a "stack" perspective. It has no database, has less moving parts. Wordpress is simple from an end user perspective. Services like medium and tumblr hide the complexity from its users. Its a trade off.
    - cocktailpeanuts 10y ago
      What you said is all correct. But my point was it's not just about "simplifying the stack". You said "the allure of a static site or a static blog is to simplify the stack;", but the thing is, not everyone thinks that way.
  - codecurve 10y ago
    Simple != easy. Wordpress and co are easy, but they're fundamentally much more complex than static sites. Jekyll is quite simple, but it can be hard to use in comparison.
sheraz 10y ago
Wow. Talk about building a rocket ship just to cross the street. Edit: My backyard is filled with old rocket ships, so I am an expert on this subject :-)
- davidwells 10y ago
  I built the rocket ship so you don't have to ™ https://www.youtube.com/watch?v=qz72V84uWWA https://www.youtube.com/watch?v=qz72V84uWWA
zokier 10y ago
Instead of hourly polling, which I find quite inelegant, couldn't you trigger creation of new schedule rule with the correct time that then triggers the publishing?
- davidwells 10y ago
  Hey zokier. Thanks for the comment. In theory yes. I could provision a new function with an exact cron schedule to run once and then tear itself down. In practice, this is a lot of extra logic to write to achieve this and would also potentially make own AWS account rather messy. The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
  - gbrits 10y ago
    How about using the new AWS Step Functions instead? This would allow a full push model while still keeping a nice and tight workflow: 1. fire off AWS Step function on new post to S3 bucket 2. read delay / schedule time in metadata from (assumable) markdown post 3. trigger AWS Step Function using the Wait State defined based on read value from 2. 4. As part of Step Function: After Wait State -> Execute Lambda to publish post. Note: 3) requires setting variable time in Wait State based on environment. Haven't read up so don't know if that's possible, but guess it should be.
    - 101km 10y ago
      This is the correct answer.
    - davidwells 10y ago
      Hey gbrits. Thanks for the comment. This looks like a great alternative solution! Haven't messed around with step functions yet but was looking into it today after I found this https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s Are you using step functions for other flows?
tomc1985 10y ago
AWS lambda and 'serverless' to replace a cronjob And paying Amazon for the privilege of doing so! Well done... golf clap
- davidwells 10y ago
  Thanks tom! I love golf! > The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier. This entire service (the 2 functions & 1 dynamo table) should run well under the AWS free tier. see https://aws.amazon.com/free/ https://aws.amazon.com/free/
  - tomc1985 10y ago
    Does this not seem excessive? Doesn't anyone care about simplicity, or elegance, or -- at minimum -- not relying on others?
    - davidwells 10y ago
      Good point. I care greatly about these things. This project greatly simplifies how we publish content in an elegant out of sight out of mind way. Before, if someone was out of town, we would need to rely on others to push the merge button on posts for us. Now, the robots do it for us!
      - tomc1985 10y ago
        While I don't know your situation your use case is understandable. It's just hard to watch such big tools serving up simple HTML with what could be a shell script or two running in cron.
    - noobiemcfoob 10y ago
      >not relying on others Avoiding AWS based solutions to "not rely on others" is like using a kerosene lamp in the center of Manhattan. You may be more self sufficient, but.... why?
intrasight 10y ago
I think this is the future of web publishing. I expect that we'll see several such frameworks for all the major "serverless" platforms.
rsingel 10y ago
All static site projects evolve into recreating WordPress.
- dexterdog 10y ago
  Because WordPress is a bloated mess when all you need is to host a fast, static site.
poyu 10y ago
I used Serverless back in the v0.5 days, picked it up again yesterday for another project. I have to say, the current version has improved vastly, from project architecture to ease of use (remember all those `json` files?!) and also documentations! Not to mention the provider support it now has. I like the idea of Serverless, and now I love it even more!
cheez 10y ago
I do this, though with Teamcity. More predictable, less AWS heavy, but cool solution especially if you already have the tools to make it easy.
rathboma 10y ago
With Jekyll you can just future-date your blog posts then trigger a republish on a daily cron. By default future-dated posts won't show up. So a daily build will publish them on the right day. If you host using Netlify you can create a build hook to just curl. You could even use Cronitor for a free cron.
- nickjj 10y ago
  This is what I do too. It works well. Although in my case, I host my Jekyll based site on my own server and just run a daily build via local cron. The workflow to post anything (present or scheduled) is nothing more than a single git push to get the new content on the server. It's as simple as can be. #serversforlife

91 points by davidwells 10y ago

fenollp 10y ago

http://explainshell.com/explain?cmd=at http://explainshell.com/explain?cmd=at Or use a CI cron running daily?

Lord_Zero 10y ago
Yeah, they said its "serverless" but it requires an AWS lambda function. I use Jekyll for my blog but I simply commit, push, and run a script that copies the files to S3. https://www.tberra.com/aws/amazon/meta/2016/11/12/the-birth-of-a-blog/ https://www.tberra.com/aws/amazon/meta/2016/11/12/the-birth-... its not scheduled but I could easily tell linux to run that command once at whatever time I wanted.
- g_delgado14 10y ago
  AWS lambda functions __are__ serverless. There's no notion of uptime, provisioning or managing servers when using lambda.
  - Lord_Zero 10y ago
    This is true, but what triggers the lambda function to run? Where does the lambda function run when its triggered?
    - climber604 10y ago
      Clearly you need to learn more about lambda https://aws.amazon.com/lambda/ https://aws.amazon.com/lambda/
    - davidwells 10y ago
      One lambda runs when github webhooks are fired: https://github.com/serverless/post-scheduler#github-webhook-architecture-overview https://github.com/serverless/post-scheduler#github-webhook-... The other lambda fires on a cron job https://github.com/serverless/post-scheduler#cron-job-architecture-overview https://github.com/serverless/post-scheduler#cron-job-archit...
    - derefr 10y ago
      "Serverless" really means "your code is run as a container with a finite lifetime on an arbitrary machine in a standardized, non-configurable PaaS machine cluster." In essence, it's a return to the "cgi-bin" model of scripting, where you write a script that only runs for the lifetime of one request, and then a web server will spin it up to respond to a relevant request and it'll terminate at the end of that request. The only differences in the modern instantiation of the concept, are that: 1. the "cgi-bin" directory is a mounted object store, common to an entire cluster; and 2. there's some fancy preloading logic that lets Lambda functions stay "hot" between calls to them, so that they can run very quickly. (Sort of like Phusion Passenger does for Ruby code, except with enough isolation to make it safe for multitenant environments. Heroku's old Alpine stack might be a better comparison.)
  - chriswarbo 10y ago
    > There's no notion of uptime, provisioning or managing servers when using lambda. There's no notion of uptime, provisioning or managing Web sites when using Web sites; but the operators of the Web site certainly have to tackle such things. Likewise, I'm sure the operators of AWS lambda are very much concerned about such things, even if the users aren't. In that sense, wordpress.com is "serverless".
    - Spivak 10y ago
      This is really just a semantic argument. You're free to call it Function as a Service or FaaS if you want. Hosted WordPress is SaaS but not FaaS.
- rhizome 10y ago
  or add a cron job to `git mv` a file from `pending` to `current` and run the commit/publish cycle.

deleted 10y ago

[deleted]

crisopolis 10y ago

Since serverless uses AWS primarily. I'm assuming if I put all my eggs in the serverless basket and Amazon has some amazing outage. I'm screwed? To which I don't understand the appeal of serverless besides having no servers lol.

davidwells 10y ago
Hello there! The serverless framework supports a number of different FAAS (function as a service) providers like IBM openwhisk, AWS lambda, & Azure functions from microsoft. See https://serverless.com/framework/docs/providers/ https://serverless.com/framework/docs/providers/ for more info. So it's up to you where your code runs =)
intrasight 10y ago
And it's still a static web site, so you're only "screwed" in terms of updates. And if you can't wait for Lambda to be restored, either run the script locally or manually hot-fix those static HTML files.

foxhop 10y ago

This solution is sort of weird to me. To me, the allure of a static site or a static blog is to simplify the stack; To bring it back to the basics. Static HTML served by a good ole-fashion web server. Adding the cognitive burden of "serverless" and/or lambda just to schedule seems to complicate something that isn't very complicated. Its almost like an invented problem. That said, long live this static site generator movement. I'm betting this movement will continue to grow which is why I'm bootstrapping https://www.remarkbox.com https://www.remarkbox.com (comments-as-a-service)

cocktailpeanuts 10y ago
I use jekyll NOT because it's simple (Actually it's 100 times more complex to use than a simple wordpress or medium) but because: 1. I can keep track changes with git 2. I can host them on Github pages for FREE 3. I own the content 4. More flexible In fact I've been really annoyed with all the boilerplate code and extensions I need to deal with in order to have a "usable" workflow set up, I almost thought about moving back to free blog services like medium, tumblr, wordpress, etc. but the only reasons I can't are the ones I mentioned above.
- foxhop 10y ago
  Jekyll is way more simple from a "stack" perspective. It has no database, has less moving parts. Wordpress is simple from an end user perspective. Services like medium and tumblr hide the complexity from its users. Its a trade off.
  - cocktailpeanuts 10y ago
    What you said is all correct. But my point was it's not just about "simplifying the stack". You said "the allure of a static site or a static blog is to simplify the stack;", but the thing is, not everyone thinks that way.
- codecurve 10y ago
  Simple != easy. Wordpress and co are easy, but they're fundamentally much more complex than static sites. Jekyll is quite simple, but it can be hard to use in comparison.

sheraz 10y ago

Wow. Talk about building a rocket ship just to cross the street. Edit: My backyard is filled with old rocket ships, so I am an expert on this subject :-)

davidwells 10y ago
I built the rocket ship so you don't have to ™ https://www.youtube.com/watch?v=qz72V84uWWA https://www.youtube.com/watch?v=qz72V84uWWA

zokier 10y ago

Instead of hourly polling, which I find quite inelegant, couldn't you trigger creation of new schedule rule with the correct time that then triggers the publishing?

davidwells 10y ago
Hey zokier. Thanks for the comment. In theory yes. I could provision a new function with an exact cron schedule to run once and then tear itself down. In practice, this is a lot of extra logic to write to achieve this and would also potentially make own AWS account rather messy. The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier.
- gbrits 10y ago
  How about using the new AWS Step Functions instead? This would allow a full push model while still keeping a nice and tight workflow: 1. fire off AWS Step function on new post to S3 bucket 2. read delay / schedule time in metadata from (assumable) markdown post 3. trigger AWS Step Function using the Wait State defined based on read value from 2. 4. As part of Step Function: After Wait State -> Execute Lambda to publish post. Note: 3) requires setting variable time in Wait State based on environment. Haven't read up so don't know if that's possible, but guess it should be.
  - 101km 10y ago
    This is the correct answer.
  - davidwells 10y ago
    Hey gbrits. Thanks for the comment. This looks like a great alternative solution! Haven't messed around with step functions yet but was looking into it today after I found this https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s https://www.youtube.com/watch?v=9MKL5Jr2zZ4&t=226s Are you using step functions for other flows?

tomc1985 10y ago

AWS lambda and 'serverless' to replace a cronjob And paying Amazon for the privilege of doing so! Well done... golf clap

davidwells 10y ago
Thanks tom! I love golf! > The hourly cron runs about 720 times a month which is far below the 1,000,000 free invocations that lambda provides under their free tier. This entire service (the 2 functions & 1 dynamo table) should run well under the AWS free tier. see https://aws.amazon.com/free/ https://aws.amazon.com/free/
- tomc1985 10y ago
  Does this not seem excessive? Doesn't anyone care about simplicity, or elegance, or -- at minimum -- not relying on others?
  - davidwells 10y ago
    Good point. I care greatly about these things. This project greatly simplifies how we publish content in an elegant out of sight out of mind way. Before, if someone was out of town, we would need to rely on others to push the merge button on posts for us. Now, the robots do it for us!
    - tomc1985 10y ago
      While I don't know your situation your use case is understandable. It's just hard to watch such big tools serving up simple HTML with what could be a shell script or two running in cron.
  - noobiemcfoob 10y ago
    >not relying on others Avoiding AWS based solutions to "not rely on others" is like using a kerosene lamp in the center of Manhattan. You may be more self sufficient, but.... why?

intrasight 10y ago

I think this is the future of web publishing. I expect that we'll see several such frameworks for all the major "serverless" platforms.

rsingel 10y ago

All static site projects evolve into recreating WordPress.

dexterdog 10y ago
Because WordPress is a bloated mess when all you need is to host a fast, static site.

poyu 10y ago

I used Serverless back in the v0.5 days, picked it up again yesterday for another project. I have to say, the current version has improved vastly, from project architecture to ease of use (remember all those `json` files?!) and also documentations! Not to mention the provider support it now has. I like the idea of Serverless, and now I love it even more!

cheez 10y ago

I do this, though with Teamcity. More predictable, less AWS heavy, but cool solution especially if you already have the tools to make it easy.

rathboma 10y ago

With Jekyll you can just future-date your blog posts then trigger a republish on a daily cron. By default future-dated posts won't show up. So a daily build will publish them on the right day. If you host using Netlify you can create a build hook to just curl. You could even use Cronitor for a free cron.

nickjj 10y ago
This is what I do too. It works well. Although in my case, I host my Jekyll based site on my own server and just run a daily build via local cron. The workflow to post anything (present or scheduled) is nothing more than a single git push to get the new content on the server. It's as simple as can be. #serversforlife