Basically, we improved a shell scripts based ETLs with lambdas. Now we barely can maintain them, and when a something breaks, it takes an engineer an inordinate

Basically, we improved a shell scripts based ETLs with lambdas. Now we barely can maintain them, and when a something breaks, it takes an engineer an inordinate amount of energy and time to get them fixed. Since we process about 0.5 to 1.5 TB of time series(IOT) I was thinking about an architecture that combine AWS kinesis and airflow