3 ms·
I'm curious what you mean by fully immutable data pipeline or DPL. Is this a data processing methodology or an actual technology?
by cakeface 9y ago
I'm curious what you mean by fully immutable data pipeline or DPL. Is this a data processing methodology or an actual technology?
- siliconc0w 9y agoAh probably sound have said 'idempotent'. Often, you want a fresh and clean resource to pop into existence, process something, persist the result someplace, and then go away. To align with the rest of AWS in terms of billing, security/IAM, networking, etc you kinda need to use the 'ec2 instance' as this resource but this causes problems so usually the customer has to create and manage another level of abstraction (like EMR, kube, mesos, ECS, what have you). It'd be real nice if the customer didn't have to do this but AWS would rather spend their engineering resources to compete with the G at beagle identification APIs or try to vertically integrate with poor clones of github. (I realize this is slightly unfair ) I know, I know - you're saying 'lambda' is the answer - won't get into that here.
- RhodesianHunter 9y agoI don't think anyone would say Lambda is the answer for anything but very quick jobs, not until they pump the available run time up to an hour.