3 ms·
Well, now you've lost me, because according to [1] the data was under 2TB (uncompressed) even as of 2015, so why did it take many days to move? I feel like ther
by icebraining 10y ago
Well, now you've lost me, because according to [1] the data was under 2TB (uncompressed) even as of 2015, so why did it take many days to move? I feel like there's something I'm missing.
[1] https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
- jedberg 10y agoYou can't just cp live data. You have to migrate it with indexes and rebuild on the fly. If you're willing to take a downtime, sure you can do it in a few days. But you still need servers to move it to. Where do those new servers come from? Are you going to buy a whole new set of servers? How long does that take? My point is that moving one datacenter to another isn't easy and there is a lot of things to think about. My point was that being in the cloud or not is barely relevant. Also that dataset isn't close to complete. It was a lot more data than that.
- mattmanser 10y agoWhy a few days? You keep saying this, why does it take a few days to move terabytes? You're not explaining it at all. Not only that, with reddit most of that data is historic and unchanging so you can pre-move it.