4 ms·
Ask HN: Any good reads, articles or books on file migration?
I’m looking to build a tool to migrate files between cloud providers (ie blob storage, s3, etc).
I’m looking to learn best practices for doing so. The volume is a few petabytes but can be done incrementally.
I was thinking some queue that handles N transfers concurrently running in a k8s cluster. Queue I was thinking Redis or Kafka. However, I’d love to build something nice and resilient
- mattpallissard 4y agoThis tool already exists and is fairly standard. No extra infra required. https://rclone.org/ https://rclone.org/
- warrenm 4y agoDon't have any articles handy (beyond the various cloud providers' offerings which you can find with a quick "s3 to s3 migration" search) But the egress and ingress fees, added-onto the middleman position your tool would be occupying sound heavy :)
- bosch_mind 4y agoBig company wallet :)
- warrenm 4y agowhy not aim for something that doesn't hurt the "Big company wallet" :)