4 ms·
What/where do you run this mythical one-analyst pipeline, though? Is that in cloud services too? Airflow? Kubeflow? Apache Beam? It sounds like you're just push
by ForHackernews 7y ago
What/where do you run this mythical one-analyst pipeline, though? Is that in cloud services too? Airflow? Kubeflow? Apache Beam? It sounds like you're just pushing the problem around.
- WoahNoun 7y agoAWS data pipelines and AWS lambda. It's cloud services the whole way down. https://aws.amazon.com/datapipeline/ https://aws.amazon.com/datapipeline/
- geggam 7y agoI saved a company 20k a month by creating a job server in AWS. Lambda isnt cheap when you start using it hard
- WoahNoun 7y agoLambda is mostly used for it's trigger functionality for data or artifacts that are created at irregular intervals. Eg, an object is uploaded to s3 which triggers a lambda which runs a copy command for that object into redshift. The kind of stuff that's well below the threshold for leaving the free tier.