4 ms·

Are you really sure?

Supermancho 6y ago
Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.
- justinram11 6y ago
  I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
- mark242 6y ago
  Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.
  - Supermancho 6y ago
    > Dynamo streams or Kinesis streams I can appreciate that. A lot more people than you might think, use Kafka or SQS because of various requirements (like that's all their infrastructure supports) and you end up with a massive "source" DB which is a duplication of all-time messages. Such is the reality of a larger byzantine organization. If it can't be audited by someone else, at their own pace, it's not approved.

Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.

justinram11 6y ago
I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
mark242 6y ago
Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.
- Supermancho 6y ago
  > Dynamo streams or Kinesis streams I can appreciate that. A lot more people than you might think, use Kafka or SQS because of various requirements (like that's all their infrastructure supports) and you end up with a massive "source" DB which is a duplication of all-time messages. Such is the reality of a larger byzantine organization. If it can't be audited by someone else, at their own pace, it's not approved.