4 ms·

I've had a lot of good experiences with ECS so far, except for their scheduled tasks system. With cron, you check /var/mail. When a scheduled task doesn't run o

I've had a lot of good experiences with ECS so far, except for their scheduled tasks system. With cron, you check /var/mail. When a scheduled task doesn't run on ECS... you're SOL.

bdcravens 9y ago
Do failed scheduled tasks not show up in the container's Cloudwatch logs?
- turdnagel 9y ago
  It's a failed CloudEvents invocation, which AFAIK means the event doesn't even hit the container/instance. There's some kind of permission that isn't set up right, but I've no idea what it is.
  - bdcravens 9y ago
    Likely an issue with ecsInstanceRole (either not assigning it to instance, or an API call that role can't access)
markplindsay 9y ago
I haven’t used ECS scheduled tasks, only the Elastic Beanstalk equivalent (which I don’t like very much). For my “worker” ECS containers, I set them up to run their tasks using an app-level scheduling library. When a task finishes, the app phones home an event to Datadog. If Datadog doesn’t get notified within a certain amount of time, I get an email. So far it’s proven to be pretty reliable. I’m sure similar functionality exists somewhere in AWS, but DD is so easy and we use it for a bunch of other stuff anyway.
emilburzo 9y ago
It sounds like you'd benefit from AWS Batch better for the scheduled tasks. After everything is up and running, all you need to do is submit a job (from a lambda function, CLI, whatever works best for you). Then, in cloudwatch event rules, you can setup a new rule that only triggers on failed jobs, which can then have a SNS/Lambda target. Or, alternatively, setup a cloudwatch alarm on the actual event rule invocations.
dguo 9y ago
I ran into this problem a few weeks ago and couldn't believe how poor the error handling is. I eventually found the FailedInvocations graph in CloudWatch, but I didn't see any way to actually find out what went wrong.
- turdnagel 9y ago
  How did you solve it?
  - dguo 9y ago
    I haven't yet. It wasn't a priority, so I stopped looking into it. I did find this forum thread though: https://forums.aws.amazon.com/thread.jspa?threadID=269884 https://forums.aws.amazon.com/thread.jspa?threadID=269884 I tried tweeting @AWSSupport, but they just referred me to the scheduled tasks docs that I had already read. I suspect the issue is that I set up my tasks to run in Fargate mode, which I didn't realize at the time was brand new. Maybe it's not compatible with scheduled tasks yet.
NathanKP 9y ago
The scheduled tasks system is built on top of Cloudwatch Events, which is asynchronous system. You would need to setup a rule in Cloudwatch Events to watch for task events that indicate that a task stopped with a nonzero exit code, or that a task was unable to be placed. It's definitely more complicated than just checking /var/mail but with the full set of features enabled you can get visibility: - If the task fails to be placed, or stops with a non zero exit code CloudWatch event rule for the task event triggers allowing you to respond to that as you see fit: email someone? trigger a PagerDuty? - If the task logging was enabled you can go to the CloudWatch log stream for the task and see the actual stdout from the container and see why it had crashed.

I've had a lot of good experiences with ECS so far, except for their scheduled tasks system. With cron, you check /var/mail. When a scheduled task doesn't run on ECS... you're SOL.

bdcravens 9y ago

Do failed scheduled tasks not show up in the container's Cloudwatch logs?

turdnagel 9y ago
It's a failed CloudEvents invocation, which AFAIK means the event doesn't even hit the container/instance. There's some kind of permission that isn't set up right, but I've no idea what it is.
- bdcravens 9y ago
  Likely an issue with ecsInstanceRole (either not assigning it to instance, or an API call that role can't access)

markplindsay 9y ago

I haven’t used ECS scheduled tasks, only the Elastic Beanstalk equivalent (which I don’t like very much). For my “worker” ECS containers, I set them up to run their tasks using an app-level scheduling library. When a task finishes, the app phones home an event to Datadog. If Datadog doesn’t get notified within a certain amount of time, I get an email. So far it’s proven to be pretty reliable. I’m sure similar functionality exists somewhere in AWS, but DD is so easy and we use it for a bunch of other stuff anyway.

emilburzo 9y ago

It sounds like you'd benefit from AWS Batch better for the scheduled tasks. After everything is up and running, all you need to do is submit a job (from a lambda function, CLI, whatever works best for you). Then, in cloudwatch event rules, you can setup a new rule that only triggers on failed jobs, which can then have a SNS/Lambda target. Or, alternatively, setup a cloudwatch alarm on the actual event rule invocations.

dguo 9y ago

I ran into this problem a few weeks ago and couldn't believe how poor the error handling is. I eventually found the FailedInvocations graph in CloudWatch, but I didn't see any way to actually find out what went wrong.

turdnagel 9y ago
How did you solve it?
- dguo 9y ago
  I haven't yet. It wasn't a priority, so I stopped looking into it. I did find this forum thread though: https://forums.aws.amazon.com/thread.jspa?threadID=269884 https://forums.aws.amazon.com/thread.jspa?threadID=269884 I tried tweeting @AWSSupport, but they just referred me to the scheduled tasks docs that I had already read. I suspect the issue is that I set up my tasks to run in Fargate mode, which I didn't realize at the time was brand new. Maybe it's not compatible with scheduled tasks yet.

NathanKP 9y ago

The scheduled tasks system is built on top of Cloudwatch Events, which is asynchronous system. You would need to setup a rule in Cloudwatch Events to watch for task events that indicate that a task stopped with a nonzero exit code, or that a task was unable to be placed. It's definitely more complicated than just checking /var/mail but with the full set of features enabled you can get visibility: - If the task fails to be placed, or stops with a non zero exit code CloudWatch event rule for the task event triggers allowing you to respond to that as you see fit: email someone? trigger a PagerDuty? - If the task logging was enabled you can go to the CloudWatch log stream for the task and see the actual stdout from the container and see why it had crashed.