3 ms·

We were pushing over a terabit of data transfer at peak, so 4-5PB per day.

atYevP 11y ago
Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!
- james_cowling 11y ago
  We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.
  - nilayp 11y ago
    Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.
    - jamwt 11y ago
      We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.
virtuallynathan 11y ago
i.e. 1Tbps? Jeez! That is crazy.
- josh2600 11y ago
  If you are moving bits from $External_DC to AWS, your best bet is to colo inside Equinix (or equivalent) and use their cross-connect to AWS service. If you have 100 fiber interconnects between you and AWS, you can move a lot of data (it becomes an interesting programming problem to figure out how to saturate that much fiber consistently). In terms of your customers getting data to you, again, fiber interconnects inside the DC are your best bet. Colo'ing is the fastest way to move bits because it relies on arteries, not capillaries, for moving bits from place to place (no last mile problem).
  - jamwt 11y ago
    Yes, balancing links and distributing work w/out congestion collapse became a really interesting and nontrivial problem.
    - josh2600 11y ago
      I can't even imagine. Did you guys get into custom network hardware or still a Juniper/Cisco shop?
      - scurvy 11y ago
        Juniper QFX from what I've heard.
james_cowling 11y ago
Btw. that was a typo, I meant over "half" a terabit of data transfer. You get into bigger numbers than this when you start talking about aggregate traffic.

Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!

james_cowling 11y ago
We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.
- nilayp 11y ago
  Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.
  - jamwt 11y ago
    We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.

virtuallynathan 11y ago

i.e. 1Tbps? Jeez! That is crazy.

josh2600 11y ago
If you are moving bits from $External_DC to AWS, your best bet is to colo inside Equinix (or equivalent) and use their cross-connect to AWS service. If you have 100 fiber interconnects between you and AWS, you can move a lot of data (it becomes an interesting programming problem to figure out how to saturate that much fiber consistently). In terms of your customers getting data to you, again, fiber interconnects inside the DC are your best bet. Colo'ing is the fastest way to move bits because it relies on arteries, not capillaries, for moving bits from place to place (no last mile problem).
- jamwt 11y ago
  Yes, balancing links and distributing work w/out congestion collapse became a really interesting and nontrivial problem.
  - josh2600 11y ago
    I can't even imagine. Did you guys get into custom network hardware or still a Juniper/Cisco shop?
    - scurvy 11y ago
      Juniper QFX from what I've heard.

james_cowling 11y ago

Btw. that was a typo, I meant over "half" a terabit of data transfer. You get into bigger numbers than this when you start talking about aggregate traffic.