3 ms·
TL;DR They switched compression from lz4 to Zstandard. The latter does more compression (and therefore requires less storage and less IO), but is slower at dec
by chrisaycock 5y ago
TL;DR They switched compression from lz4 to Zstandard.
The latter does more compression (and therefore requires less storage and less IO), but is slower at decompression. Results show that query (read) performance did not actually change, whereas write operations needed only half as much time. Storage also saved ~20% space. So it was a win-win all around.
- LolWolf 5y agoThanks ! I love some of the articles, but in this case I definitely went the “I’m happy for you or sad it happened, but I ain’t about to read all of that” route.
- jandrese 5y agoMost of the savings seemed to come from freeing up enough headroom in each drive to prevent block collating slowdowns on write. This smells like a temporary workaround to me, as data tends to grow over time. It probably saved them from having to buy more storage this quarter, but it is a one time savings.
- turbocon 5y agoTo the contrary! This decreases their storage need by ~20% which will increase their cost savings over time. Yes they pushed off increasing their storage footprint in the short term but in the long term they decreased their rate of total storage increase.
- stjohnswarts 5y agoIt saves them on number of drives. Of course they can always buy more drive capacity on AWS if a certain percentage of their drive pools starts to hit 80% capacity, it's not a one time savings unless AWS stops charging them and lets them use their services for free. It would be a recurring savings.