calebtv

4 ms·

1. 1.▲
   by calebtv 1y ago

   This will hopefully be our last pivot :) BuildFlow never took off and we're still supporting the companies who use the LaunchFlow Python SDK.
2. 2.▲
   by calebtv 1y ago

   Yeah Terraform does have some sharp edges, but we chose it because it's the most widely used IaC language and lets you review any changes the agent wants to make before actually applying them in your cloud. One issue with an agent maki
3. 3.▲
   by calebtv 1y ago

   We do want to support clouds beyond AWS, GCP, and Azure. That's one of the reasons we choose Terraform since it can be extendable to any cloud, and also why we plan on focusing on Kubernetes next.
4. 4.▲
   Show HN: Open-source deployment tool for AWS Lambda(docs.launchflow.com)

   3 points by calebtv 2y ago|0 comments
5. 5.▲
   Switch Between Serverless and Dedicated VMs on GCP(launchflow.com)

   1 points by calebtv 2y ago|0 comments
6. 6.▲
   by calebtv 4y ago

   Thanks for the kind words Robert! Our experience with Ray has been great so far, we're excited to see how we can use ray to help improve stream processing.
7. 7.▲
   by calebtv 4y ago

   I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...
8. 8.▲
   by calebtv 4y ago

   I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...
9. 9.▲
   by calebtv 4y ago

   Good question, I would say we're more focused on being a data pipeline engine as opposed to workflow orchestration. So you could use something like Airflow or Dagster to trigger your BuildFlow pipeline.
10. 10.▲
    by calebtv 4y ago

    Thanks! Currently you can't, right now your only option is to use our ray runner. But we have talked about supporting different runner options similar to how Beam can be run on Spark, Dataflow, etc. And ultimately it would be nice if f
11. 11.▲
    by calebtv 4y ago

    We don't support any snapshotting or checkpointing directly in BuildFlow at the moment, but these are great features we should support. But we do have some fault tolerance baked into our I/O operations. Specifically for Google Clo
12. 12.▲
    by calebtv 4y ago

    Thanks! These are all great questions, apologies for the wall of text 1. We're definitely more of a generic streaming framework. But I could see ML being one of those use cases as well. Why Ray? One of our main drivers was how "py
13. 13.▲
    by calebtv 4y ago

    Great question! We actually looked at using the workflow abstraction for batch processing in our runner, but ultimately didn't because it was still in alpha (we use the dataset API for batch flows). I think one area where we differ is
14. 14.▲
    by calebtv 4y ago

    All of our processing is done via Ray ( https://www.ray.io/ ). Our early benchmarks are about 5k mesesages per second on a single 4 core VM, but we believe we can increase the with some more optimizations. This bench mark was
15. 15.▲
    by calebtv 4y ago

    One other thing I should mention that's relevent, we do also have a class abstraction instead of a decorator: https://github.com/launchflow/buildflow/blob/main/buildflow/... This can help with
16. 16.▲
    by calebtv 4y ago

    More of a higher level library like Beam, and I could see it being plugged into a Prefect workflow.
17. 17.▲
    by calebtv 4y ago

    Thanks! The process function runs as a Ray Actor ( https://docs.ray.io/en/latest/ray-core/actors.html ). So we have the same serialization requirements as Ray ( https://docs.ray.io/en/latest
18. 18.▲
    Launch HN: BuildFlow (YC W23) – The FastAPI of data pipelines

    104 points by calebtv 4y ago|35 comments