4 ms·
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over
by ericbarrett 4y ago
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
- Xorlev 4y agoPretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds. I recall learning the technique from Tumblr's eng blog. https://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations https://engineering.tumblr.com/post/7658008285/efficiently-c...
- evanelias 4y agoI wrote that Tumblr eng blog post, glad to see it's still making the rounds! I later joined FB's mysql team a few years after that, although I can't quite remember if FB was still using pigz by that time. (also, hi Eric!) Separately, at Tumblr I vaguely remember examining some alternative to pigz that was consistently faster at the time (11 years ago) because pigz couldn't parallelize decompression. Can't quite remember the name of the alternative, but it had licensing restrictions which made it less attractive than pigz. Edit: the old fast alternative I was thinking of is qpress, formerly hosted at http://www.quicklz.com/ http://www.quicklz.com/ but that's no longer online. Googling it now, there are some mirrors and also looks like Percona tools used/bundled it. Not sure if they still do or if they've since switched to zstd.
- Xorlev 4y agoSmall world! Thanks for writing that, it was a really clever way to do it and saved me a bunch of time. :)
- antisthenes 4y agoSame, except we were at a small e-commerce boutique running Magento circa 2011-2013. SQL backups were simply a bash script using Pigz, running on a cron job. Simple times!
- mackman 4y agoHey Eric! Hope you’re well!