4 ms·

I have written this reply and started over 3 times -- hopefully this one sticks. I really feel like the author boxed himself into a solution by faulty reasonin

I have written this reply and started over 3 times -- hopefully this one sticks. I really feel like the author boxed himself into a solution by faulty reasoning. 1) Cron jobs are not hard to set up. Being able to control these things on one or more server is just part of proper server deployment. 2) If it is not part of the application but seems to have to do with information about users who used the application in trial then maybe it should be part of the application? If you are deploying a application and you require to know information about users who trials are going to end I sure would bet that is part of the application and should be a trigger coded into the application itself. 3) I am shocked that you are already did not have a central spot for running time interval code and maintenance on your systems. At a minimum there should have been backups and reports generated on your mongodb instances. If these functions were already built into the application then then the same generic task system you used to run that is likely where you wanted to run this new functionality. tl;dr Not knowing much about the application or its architecture I feel like the author painted themselves into a corner to justify using some AWS tools.

jasonmp85 10y ago
Having just written a bunch of scripts to populate a Google Sheet with some KPI data for the business folks, I _completely_ agree. At first I wanted to put my scripts in Lambda, put an API Gateway in front, and have them be callable by the Google Sheet on a schedule. This seemed "pure" because I didn't really _need_ independent resources to do the task. The sheet could just update itself using an on-demand server! Then I realized there was a 30-second max execution time. My scripts, in serial, couldn't hit that. I'd need to rewrite them all to be parallel. Also, they were a bunch of bash/jq/curl I'd just thrown together. I'd have to write a Python wrapper to shell out and handle streams (ugh). At this point, I gave up and fell back to the advice of an (ex-Heroku) coworker: why not just run them in a cron job and put the data in a Heroku DB and use Heroku Dataclips to update the Google Sheet. Sure I have to "have" an EC2 instance and a Heroku PG box, but they're both so small they're either free or essentially so. After cloning my script repos, everything worked right away. Sometimes it's better just to do the "inelegant" thing and move on.
- teej 10y ago
  Embarrassingly, I completed practically the exact same project but stuck it through with Lambda/API Gateway and just have it pull the precomputed data from a database. I wish I had known about Heroku Dataclips earlier! The build/test/deploy cycle for a Google Sheets add-on is a total nightmare.
- CWuestefeld 10y ago
  Then I realized there was a 30-second max execution time. Perhaps you're talking about a limitation on your spreadsheet, but the max execution time for a Lambda function is configurable up to 300 seconds.
  - jasonmp85 10y ago
    I believe it may have been a limitation of the API Gateway + Lambda thing. Just checked the docs again and it says there is an "Integration Timeout" as follows: > 30 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations. So not Lambda alone, but in conjunction with the API Gateway.
- colechristensen 10y ago
  On the other side of the coin, a few months ago I was annoyed by getting a redshift alert one too many times during maintenance that I spent a couple of hours on a Saturday writing a ~20 line Lambda project that took the notifications from Redshift and posted them in a Slack channel. I didn't have to worry about setting up automation for launching an ec2 instance, monitoring it, paying for it, or any of the complex trouble for what ended up being a single python function that cost abotu $0.10 per month to run.
jwilliams 10y ago
I don't really take issue with running this on AWS Lambda. Seems as good a place as any. You're right that there are other straightforward options, but they do have downsides. Worst case is you have mismatch of workloads (e.g. a rogue cronjob affecting another service). Or you end up with resources that are largely idle. That said, I agree with your unease on the application side. If you had a dozen (or many dozen) of these you've got pieces and parcels of logic all over the place. I think some architecture/style changes could be mitigations here. A shared library might be one.
dvdhnt 10y ago
I agree. It seems like once he decided that > The cron job that we needed to write was unrelated to our application code, so whilst we could have put the functionality in there it seemed like the wrong place. he never looked back at existing architecture and invented a challenge for himself.

jasonmp85 10y ago

Having just written a bunch of scripts to populate a Google Sheet with some KPI data for the business folks, I _completely_ agree. At first I wanted to put my scripts in Lambda, put an API Gateway in front, and have them be callable by the Google Sheet on a schedule. This seemed "pure" because I didn't really _need_ independent resources to do the task. The sheet could just update itself using an on-demand server! Then I realized there was a 30-second max execution time. My scripts, in serial, couldn't hit that. I'd need to rewrite them all to be parallel. Also, they were a bunch of bash/jq/curl I'd just thrown together. I'd have to write a Python wrapper to shell out and handle streams (ugh). At this point, I gave up and fell back to the advice of an (ex-Heroku) coworker: why not just run them in a cron job and put the data in a Heroku DB and use Heroku Dataclips to update the Google Sheet. Sure I have to "have" an EC2 instance and a Heroku PG box, but they're both so small they're either free or essentially so. After cloning my script repos, everything worked right away. Sometimes it's better just to do the "inelegant" thing and move on.

teej 10y ago
Embarrassingly, I completed practically the exact same project but stuck it through with Lambda/API Gateway and just have it pull the precomputed data from a database. I wish I had known about Heroku Dataclips earlier! The build/test/deploy cycle for a Google Sheets add-on is a total nightmare.
CWuestefeld 10y ago
Then I realized there was a 30-second max execution time. Perhaps you're talking about a limitation on your spreadsheet, but the max execution time for a Lambda function is configurable up to 300 seconds.
- jasonmp85 10y ago
  I believe it may have been a limitation of the API Gateway + Lambda thing. Just checked the docs again and it says there is an "Integration Timeout" as follows: > 30 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations. So not Lambda alone, but in conjunction with the API Gateway.
colechristensen 10y ago
On the other side of the coin, a few months ago I was annoyed by getting a redshift alert one too many times during maintenance that I spent a couple of hours on a Saturday writing a ~20 line Lambda project that took the notifications from Redshift and posted them in a Slack channel. I didn't have to worry about setting up automation for launching an ec2 instance, monitoring it, paying for it, or any of the complex trouble for what ended up being a single python function that cost abotu $0.10 per month to run.

jwilliams 10y ago

I don't really take issue with running this on AWS Lambda. Seems as good a place as any. You're right that there are other straightforward options, but they do have downsides. Worst case is you have mismatch of workloads (e.g. a rogue cronjob affecting another service). Or you end up with resources that are largely idle. That said, I agree with your unease on the application side. If you had a dozen (or many dozen) of these you've got pieces and parcels of logic all over the place. I think some architecture/style changes could be mitigations here. A shared library might be one.

dvdhnt 10y ago

I agree. It seems like once he decided that > The cron job that we needed to write was unrelated to our application code, so whilst we could have put the functionality in there it seemed like the wrong place. he never looked back at existing architecture and invented a challenge for himself.