10 ms·

Migrating 23TB from S3 to B2 in 7 hours

192 points by CherryJimbo 8y ago

paddor 7y ago
Can I ask why ZIP? Isn’t it quite heavy on CPU at a not-that-good compression rate? I’m thinking of LZ4 or Zstandard instead.
- CherryJimbo 7y ago
  We had to pick something that was "good enough" for compression time/size, as well as easy for our customers to download/view if they wish, on any OS. Zip being supported in every popular operating system, and the average user using Windows being able to right click -> unzip, was the primary reason for choice. There are of course significantly faster and more efficient compression formats like LZ4 which would be ideal if we were solely using the data internally in managed environments, but we offer these backups as downloads to our users, some of which aren't very technically inclined and still need to be able to access the files easily.
howiroll 8y ago
I’ve heard Backblaze has only one region. Is that true?
- CherryJimbo 8y ago
  Unfortunately... yes. They have Europe coming "really soon" when we were talking to them which we're excited about, but it's one location in the west coast of the US currently.
- chx 8y ago
  They said they use Cloudflare as well...
  - CherryJimbo 8y ago
    We do use Cloudflare, but a lot of the instance backups we store are multiple GBs - Cloudflare doesn't cache those. Not to mention uploads from regions like Singapore can be very slow all the way to the US. It's not a deal-breaker for us, but we're very much looking forward to when they can support more regions.
andrewstuart 8y ago
@ USD$0.05 cents per gigabyte out from S3, that's about USD $1150 to transfer 23TB is that right? https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ I'd be pretty nervous about hitting the delete button after the data transfer.
- CherryJimbo 8y ago
  Almost double that, as we didn't hit the tier for the lower $0.05 pricing. We also used some of their more expensive locations for some storage too, like Singapore - the 23TB was spread between different regions.
  - qaq 8y ago
    Ain't aws great would've being like what $250 on DO
    - Operyl 8y ago
      And then you also get to deal with DO losing entire clusters of Spaces, for a week+ at a time with no timely status updates. That was great ..
    - scrollaway 8y ago
      You're not buying the same thing. Comparing AWS to DO is like comparing your home printer-scanner with the Hubble telescope. Yes, one of them is more expensive.
      - deleted 8y ago
        [deleted]
      - lultimouomo 8y ago
        Really? I'm not saying AWS doesn't offer anything over bare servers, but do people think it's a Hubble to their home printers?
        8y ago
sanxchit 8y ago
Great writeup on data migrations. I was wondering whether you did a comparison for this method vs using AWS snowball[1] to export S3 Data and B2 Fireball[2] to ingest it. [1] - https://docs.aws.amazon.com/snowball/latest/ug/create-export-job-steps.html https://docs.aws.amazon.com/snowball/latest/ug/create-export... [2] - https://help.backblaze.com/hc/en-us/articles/360001918654 https://help.backblaze.com/hc/en-us/articles/360001918654
- CherryJimbo 8y ago
  We looked briefly at the snowball and fireball, but wanted to do this as quickly as possible, whilst keeping the process entirely transparent to our users. It was also an excuse for our team to get intimately familiar with the B2 API, since it's not compatible with S3. If we were to consider another large migration like this, physical media would probably be the way to go.
  - late2part 8y ago
    I evaluated moving 2PB with snowball v. putting in 10g/100g links. The issue with snowball ( i started a company that did what snowball did and shut it down (failed )) and other fedex/RAID solutions is that you have 3 transfers. You think the LAN transfer will be quick - but you're generally rate-limited by the systems more than you are by the bandwidth-delay product. If you're in a high traffic DC area, it's pretty easy to get temp bw or install circuits to carry that. 10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping. Those numbers aren't real but they're close. So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it. pigz/scp/zstd works extremely fast in line. In your case you're pulling from S3 to another object store. I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys. Now I work for a large cloud company and we still run into elasticity. To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print). Worked great. Good for you for sharing this project, very cool.
kbowman 8y ago
Did you compare it against Wasabi?
- CherryJimbo 8y ago
  We did not. Wasabi looks interesting - thanks!
  - chime 8y ago
    We have about 125TB on Wasabi. Costs about 650/mo. Their performance for large files is great.
    - metildaa 8y ago
      Haven't they had stability issues? I know it was troublesome for some Mastodon instances.
      - chime 8y ago
        Once so far in last 8mo. It wasn’t a big issue in our use case, especially since there was no data loss.
  - manigandham 8y ago
    I also recommend Wasabi. Even cheaper than B2 and they follow the S3 API so all existing tools still work. They also have different billing models with a free egress plan if you aren't downloading data that often.
late2part 8y ago
What's the monthly storage cost on that?
- StavrosK 8y ago
  $115.
  - late2part 8y ago
    https://imgur.com/gallery/lqKlotB https://imgur.com/gallery/lqKlotB
Proven 8y ago
That's 912 MB/s, most servers can handle that with ease. A commercial pay-as-you-go data sync utility could do the job (copy, validate, parallelize) for a couple of bucks. For a one-off migration that can be done just as well with existing tools I wouldn't waste engineering time on creating a custom solution.
Bombthecat 8y ago
Are they still one data center only?
- BartBoch 8y ago
  It seems that they have two DCs now, not sure if the second is used for redundancy though.
toomuchtodo 8y ago
Would you consider open sourcing the micro service you wrote to perform the migration? I could see it being helpful to others interested in migrating from S3 to B2.
- toomuchtodo 7y ago
  This request can be disregarded. I’m going to explore extending s3proxy for the same purpose (migration and backfill of disparate object storage systems through an abstraction layer).
dgemm 8y ago
If this is purely backup data, wouldn't glacier be a better fit than either S3 or B2? Glacier is already cheaper than B2 and has the advantage of storing data redundantly across datacenters. And glacier deep archive is 4 times cheaper than that. Full disclosure: AWS employee
- mrtesthah 8y ago
  Restoring anything from glacier is a pain in the ass due to all the arbitrary restrictions that apply.
- snissn 8y ago
  No - the pricing on glacier makes it a steaming pile of shit
  - dgemm 8y ago
    It's built for off-site backup type workloads, where you write once and read hopefully never. You are probably using it wrong.
    - deleted 8y ago
      [deleted]
  - camtarn 8y ago
    I was pleasantly surprised to see that the previous rather crazy retrieval pricing model (price based on the peak retrieval bandwidth you used at any time in the current month) was replaced by a straight price-per-GB model a couple years back. That made my life a lot easier when pricing up Glacier vs B2 for a client.
- CherryJimbo 8y ago
  They're "backups" in the sense of storing customer data, but realistically they're more like instance snapshots for various customer game servers. They're created and restored multiple times a day for every customer as they hop between games (accessed very frequently), so Glacier wouldn't be a good fit.
cavisne 8y ago
I'm guessing the payback time on this is fairly long by the time you factor in the cost of moving all the backups. S3 One Zone is pretty competitive against Backblaze. Cloudflare is betting on most of their customers serving HTML not large uncachable blobs, that "Bandwidth Alliance" will disappear pretty quickly at any sort of scale.
- CherryJimbo 8y ago
  I can't speak for Cloudflare, but we've talked to them pretty extensively about the whole project, and even did a case study about our use-case with the bandwidth alliance, and as we switched cloud providers. Things may change in the future of course, but they very much encouraged what we were doing. https://www.cloudflare.com/case-studies/nodecraft-bandwidth-alliance/ https://www.cloudflare.com/case-studies/nodecraft-bandwidth-...
ryanmarsh 8y ago
The stated reason for moving to B2 was pricing. No breakdown of cost was given however. I’m not sure how they arrived at that conclusion.
stevefan1999 8y ago
I mean, the best way to migrate this huge amount of data was still has had to be using a physical migration service like AWS Snowmobile[0], right? [0]: https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
vviktor 8y ago
Wouldn't it be simpler if you gradually moved? https://martinfowler.com/bliki/ParallelChange.html https://martinfowler.com/bliki/ParallelChange.html I understand your main concern for moving was pricing. Developer hours also cost money. It seems like you had to invest much more developer hours than you would have if you gradually moved over the course of one month or so. (probably a week?)
Can_Not 8y ago
I assume I'm not the only one who has never heard of B2 before now, so I'll summarize everything I've found out: https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html It appears to be an S3 clone. Competing via lower price and less micro charges. Highlights compared to Digital Ocean and Wasabi: DO/Wasabi: minimum $5/month, but great deals compared to AWS/GCE otherwise. B2: First 10GB storage free (probably not including bandwidth). For a side project or startup looking for its first storage option, B2 seems compelling. But something important: is it a drop in replacement? Is the api accessible on your platform? https://help.backblaze.com/hc/en-us/articles/218513487-Is-the-B2-Cloud-Storage-API-Compatible-with-Amazon-S3-?mobile_site=true https://help.backblaze.com/hc/en-us/articles/218513487-Is-th... I don't have a clear answer right now, but when I get closer to deploying my current side project, it's something I'll take a deeper look into.
- iansltx 7y ago
  Their API is not S3 compatible. There are a bunch of client libraries on various platforms for it though. They have 1GB free of outgoing bandwidth per day. If you put them behind Cloudflare then you effectively get your bandwidth for free. So half a cent per GB per month is all you way. Plus your time building out the integration instead of S3.

192 points by CherryJimbo 8y ago

Can I ask why ZIP? Isn’t it quite heavy on CPU at a not-that-good compression rate? I’m thinking of LZ4 or Zstandard instead.

CherryJimbo 7y ago
We had to pick something that was "good enough" for compression time/size, as well as easy for our customers to download/view if they wish, on any OS. Zip being supported in every popular operating system, and the average user using Windows being able to right click -> unzip, was the primary reason for choice. There are of course significantly faster and more efficient compression formats like LZ4 which would be ideal if we were solely using the data internally in managed environments, but we offer these backups as downloads to our users, some of which aren't very technically inclined and still need to be able to access the files easily.

howiroll 8y ago

I’ve heard Backblaze has only one region. Is that true?

CherryJimbo 8y ago
Unfortunately... yes. They have Europe coming "really soon" when we were talking to them which we're excited about, but it's one location in the west coast of the US currently.
chx 8y ago
They said they use Cloudflare as well...
- CherryJimbo 8y ago
  We do use Cloudflare, but a lot of the instance backups we store are multiple GBs - Cloudflare doesn't cache those. Not to mention uploads from regions like Singapore can be very slow all the way to the US. It's not a deal-breaker for us, but we're very much looking forward to when they can support more regions.

andrewstuart 8y ago

@ USD$0.05 cents per gigabyte out from S3, that's about USD $1150 to transfer 23TB is that right? https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ I'd be pretty nervous about hitting the delete button after the data transfer.

CherryJimbo 8y ago
Almost double that, as we didn't hit the tier for the lower $0.05 pricing. We also used some of their more expensive locations for some storage too, like Singapore - the 23TB was spread between different regions.
- qaq 8y ago
  Ain't aws great would've being like what $250 on DO
  - Operyl 8y ago
    And then you also get to deal with DO losing entire clusters of Spaces, for a week+ at a time with no timely status updates. That was great ..
  - scrollaway 8y ago
    You're not buying the same thing. Comparing AWS to DO is like comparing your home printer-scanner with the Hubble telescope. Yes, one of them is more expensive.
    - deleted 8y ago
      [deleted]
    - lultimouomo 8y ago
      Really? I'm not saying AWS doesn't offer anything over bare servers, but do people think it's a Hubble to their home printers?
      - 8y ago

sanxchit 8y ago

Great writeup on data migrations. I was wondering whether you did a comparison for this method vs using AWS snowball[1] to export S3 Data and B2 Fireball[2] to ingest it. [1] - https://docs.aws.amazon.com/snowball/latest/ug/create-export-job-steps.html https://docs.aws.amazon.com/snowball/latest/ug/create-export... [2] - https://help.backblaze.com/hc/en-us/articles/360001918654 https://help.backblaze.com/hc/en-us/articles/360001918654

CherryJimbo 8y ago
We looked briefly at the snowball and fireball, but wanted to do this as quickly as possible, whilst keeping the process entirely transparent to our users. It was also an excuse for our team to get intimately familiar with the B2 API, since it's not compatible with S3. If we were to consider another large migration like this, physical media would probably be the way to go.
- late2part 8y ago
  I evaluated moving 2PB with snowball v. putting in 10g/100g links. The issue with snowball ( i started a company that did what snowball did and shut it down (failed )) and other fedex/RAID solutions is that you have 3 transfers. You think the LAN transfer will be quick - but you're generally rate-limited by the systems more than you are by the bandwidth-delay product. If you're in a high traffic DC area, it's pretty easy to get temp bw or install circuits to carry that. 10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping. Those numbers aren't real but they're close. So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it. pigz/scp/zstd works extremely fast in line. In your case you're pulling from S3 to another object store. I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys. Now I work for a large cloud company and we still run into elasticity. To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print). Worked great. Good for you for sharing this project, very cool.

kbowman 8y ago

Did you compare it against Wasabi?

CherryJimbo 8y ago
We did not. Wasabi looks interesting - thanks!
- chime 8y ago
  We have about 125TB on Wasabi. Costs about 650/mo. Their performance for large files is great.
  - metildaa 8y ago
    Haven't they had stability issues? I know it was troublesome for some Mastodon instances.
    - chime 8y ago
      Once so far in last 8mo. It wasn’t a big issue in our use case, especially since there was no data loss.
- manigandham 8y ago
  I also recommend Wasabi. Even cheaper than B2 and they follow the S3 API so all existing tools still work. They also have different billing models with a free egress plan if you aren't downloading data that often.

late2part 8y ago

What's the monthly storage cost on that?

StavrosK 8y ago
$115.
- late2part 8y ago
  https://imgur.com/gallery/lqKlotB https://imgur.com/gallery/lqKlotB

Proven 8y ago

That's 912 MB/s, most servers can handle that with ease. A commercial pay-as-you-go data sync utility could do the job (copy, validate, parallelize) for a couple of bucks. For a one-off migration that can be done just as well with existing tools I wouldn't waste engineering time on creating a custom solution.

Bombthecat 8y ago

Are they still one data center only?

BartBoch 8y ago
It seems that they have two DCs now, not sure if the second is used for redundancy though.

toomuchtodo 8y ago

Would you consider open sourcing the micro service you wrote to perform the migration? I could see it being helpful to others interested in migrating from S3 to B2.

toomuchtodo 7y ago
This request can be disregarded. I’m going to explore extending s3proxy for the same purpose (migration and backfill of disparate object storage systems through an abstraction layer).

dgemm 8y ago

If this is purely backup data, wouldn't glacier be a better fit than either S3 or B2? Glacier is already cheaper than B2 and has the advantage of storing data redundantly across datacenters. And glacier deep archive is 4 times cheaper than that. Full disclosure: AWS employee

mrtesthah 8y ago
Restoring anything from glacier is a pain in the ass due to all the arbitrary restrictions that apply.
snissn 8y ago
No - the pricing on glacier makes it a steaming pile of shit
- dgemm 8y ago
  It's built for off-site backup type workloads, where you write once and read hopefully never. You are probably using it wrong.
  - deleted 8y ago
    [deleted]
- camtarn 8y ago
  I was pleasantly surprised to see that the previous rather crazy retrieval pricing model (price based on the peak retrieval bandwidth you used at any time in the current month) was replaced by a straight price-per-GB model a couple years back. That made my life a lot easier when pricing up Glacier vs B2 for a client.
CherryJimbo 8y ago
They're "backups" in the sense of storing customer data, but realistically they're more like instance snapshots for various customer game servers. They're created and restored multiple times a day for every customer as they hop between games (accessed very frequently), so Glacier wouldn't be a good fit.

cavisne 8y ago

I'm guessing the payback time on this is fairly long by the time you factor in the cost of moving all the backups. S3 One Zone is pretty competitive against Backblaze. Cloudflare is betting on most of their customers serving HTML not large uncachable blobs, that "Bandwidth Alliance" will disappear pretty quickly at any sort of scale.

CherryJimbo 8y ago
I can't speak for Cloudflare, but we've talked to them pretty extensively about the whole project, and even did a case study about our use-case with the bandwidth alliance, and as we switched cloud providers. Things may change in the future of course, but they very much encouraged what we were doing. https://www.cloudflare.com/case-studies/nodecraft-bandwidth-alliance/ https://www.cloudflare.com/case-studies/nodecraft-bandwidth-...

ryanmarsh 8y ago

The stated reason for moving to B2 was pricing. No breakdown of cost was given however. I’m not sure how they arrived at that conclusion.

stevefan1999 8y ago

I mean, the best way to migrate this huge amount of data was still has had to be using a physical migration service like AWS Snowmobile[0], right? [0]: https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/

vviktor 8y ago

Wouldn't it be simpler if you gradually moved? https://martinfowler.com/bliki/ParallelChange.html https://martinfowler.com/bliki/ParallelChange.html I understand your main concern for moving was pricing. Developer hours also cost money. It seems like you had to invest much more developer hours than you would have if you gradually moved over the course of one month or so. (probably a week?)

Can_Not 8y ago

I assume I'm not the only one who has never heard of B2 before now, so I'll summarize everything I've found out: https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html It appears to be an S3 clone. Competing via lower price and less micro charges. Highlights compared to Digital Ocean and Wasabi: DO/Wasabi: minimum $5/month, but great deals compared to AWS/GCE otherwise. B2: First 10GB storage free (probably not including bandwidth). For a side project or startup looking for its first storage option, B2 seems compelling. But something important: is it a drop in replacement? Is the api accessible on your platform? https://help.backblaze.com/hc/en-us/articles/218513487-Is-the-B2-Cloud-Storage-API-Compatible-with-Amazon-S3-?mobile_site=true https://help.backblaze.com/hc/en-us/articles/218513487-Is-th... I don't have a clear answer right now, but when I get closer to deploying my current side project, it's something I'll take a deeper look into.

iansltx 7y ago
Their API is not S3 compatible. There are a bunch of client libraries on various platforms for it though. They have 1GB free of outgoing bandwidth per day. If you put them behind Cloudflare then you effectively get your bandwidth for free. So half a cent per GB per month is all you way. Plus your time building out the integration instead of S3.