24 ms·
We saved millions in SSD costs by upgrading our filesystem
- deleted 5y ago[deleted]
- paladin314159 5y agoWe switched from LZ4 to Zstd in almost all of our compression use cases to great effect. Reducing the data on disk or over the network is a huge win with only a minor loss in decompression speed (using the appropriate level of Zstd). E.g. data in Kafka: https://amplitude.engineering/reducing-kafka-costs-with-z-tandard-88e0e489850f https://amplitude.engineering/reducing-kafka-costs-with-z-ta...
- wanderer2323 5y ago... from ZFS (lz4) to ZFS 2.x (Zstandard).
- deleted 5y ago[deleted]
- chungy 5y agoIt is an upgrade, but don't mistake ZFS 2.x as making zstandard mandatory. The default for compression=on is still lz4.
- ddlutz 5y agoI wonder how well Postgres is up to the task of analytical queries? Most people use Postgres for OLTP, maybe they are running some version of it that uses a column store?
- jhk727 5y agoPostgres is not designed for OLAP, but you can push it a lot farther than one would expect with the correct schema and indexing strategy. See https://heap.io/blog/running-10-million-postgresql-indexes-in-production https://heap.io/blog/running-10-million-postgresql-indexes-i... for a little more detail about how we schematize for distributed OLAP queries on Postgres at scale.
- __s 5y agoThey use Citus: https://www.citusdata.com/customers/heap https://www.citusdata.com/customers/heap Who recently iterated on cstore_fdw to create columnar: https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-compression-for-postgres https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-... But I don't think Heap's using columnar
- jhk727 5y agoWe aren't using cstore_fdw, though we've looked into it in the past. cstore tables don't support deletes or updates, and we still rely on updates for some key parts of our write pipeline. Additionally, we rely heavily on btree partial indexes, while cstore tables only support skip indexes.
- KennyBlanken 5y agoSince the title is clickbaity: they had issues with ZFS due to too-high a blocksize and too-high a filesystem utilization, so they upgraded to ZFS 2 for the Zstandard compression and saw an improvement.
- ziddoap 5y agoIs it clickbaity if they actually did save millions in SSD costs? What would you suggest as a non-clickbaity title? Just "Upgrading Our Filesystem" leaves out the important parts (the why and the results). A lot of the time I agree that titles can be quite clickbaity. But this one doesn't really seem to be... At least to me. The company upgraded their filesystem and it saved them a bunch of money. Title feels appropriate. If the title were "Top ways to save millions on SSDs!" or similar, I'd wholeheartedly agree.
- jaclaz 5y agoBut: >Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes) If 21% reduction is "millions" they should be spending in excess of 10 millions (each what? week/month/year?) provided that there is a linear correlation between storage usage and (failed and needing to be replaced?) SSD costs.
- slownews45 5y agoThere's a reason I click to the comments first most of the time.
- chrisaycock 5y agoTL;DR They switched compression from lz4 to Zstandard. The latter does more compression (and therefore requires less storage and less IO), but is slower at decompression. Results show that query (read) performance did not actually change, whereas write operations needed only half as much time. Storage also saved ~20% space. So it was a win-win all around.
- LolWolf 5y agoThanks ! I love some of the articles, but in this case I definitely went the “I’m happy for you or sad it happened, but I ain’t about to read all of that” route.
- jandrese 5y agoMost of the savings seemed to come from freeing up enough headroom in each drive to prevent block collating slowdowns on write. This smells like a temporary workaround to me, as data tends to grow over time. It probably saved them from having to buy more storage this quarter, but it is a one time savings.
- turbocon 5y agoTo the contrary! This decreases their storage need by ~20% which will increase their cost savings over time. Yes they pushed off increasing their storage footprint in the short term but in the long term they decreased their rate of total storage increase.
- stjohnswarts 5y agoIt saves them on number of drives. Of course they can always buy more drive capacity on AWS if a certain percentage of their drive pools starts to hit 80% capacity, it's not a one time savings unless AWS stops charging them and lets them use their services for free. It would be a recurring savings.
- kevvok 5y agoI’m not sure how much I buy their claim that ZFS being COW is worse for SSDs given SSDs themselves are COW to allow for wear-leveling because they have to erase whole blocks and rewrite them if even one byte is changed
- b112 5y agoThey say their fs block size is 64kb. How large are SSD blocks?
- jeffbee 5y agoSSD erase blocks vary, could be anywhere from 64KiB to several MiB. Their logical blocks (visible to the host operating system) are usually either 512 or 4096 bytes.
- wtallis 5y agoNAND flash erase blocks passed 1MB a long time ago. For mainstream TLC NAND, 16 to 24 MB is currently typical, and QLC NAND has gone as high as 48 and 96 MB. NAND page sizes are usually 16kB, but often with support for faster partial page programming for the sake of 4kB operations. Logical block sizes of 512 bytes are purely a fiction for the sake of compatibility and SSDs don't actually track allocations at that granularity. They do track things at 4kB granularity even though that's not quite large enough to be a good fit for today's flash.
- namibj 5y agoWhere does one find information on erase granularity, and, like, doing the optimizations offered by NVMe ZNS on a "normal" SSD? I assume if you throw a sufficiently uncomplicated ZNS-like workload at them, they don't try to ruin your efforts? I guess arguably this comes from the lack of availability of ZNS drives, especially ones that aren't outcompeted by a Samsung 980 series drive on workloads that don't exceed the 980's warranty's 600 drive writes over 5 years. And depending on what you're doing, you may even use such drives for the IO-heavy load and retire them after 90% of their TBW to something like fileserver duty.
- londons_explore 5y agoPostgres is copy on write. (A modified record is copied to a new page when written, and the old record left in place till a vacuum). ZFS is Copy-on-write (One byte written in a block requires the whole block be rewritten, with the old one scheduled for reclamation). The underlying SSD wear levelling algorithm is Copy on Write (Writing a single byte involves writing the data from the page to a new block, and then erasing the old one sometime later) That means a tiny 1 byte modification to a postgres record involves creating many new unnecessary copies of a lot of data... I imagine that if the three layers could be combined, dramatic performance benefits could happen, since written data might go down by an order of magnitude at least....
- tmikaeld 5y agoIf anything, it would drastically increase the lifetime of the SSDs.
- jandrese 5y agoSSDs already do copy-on-write internally, so this doesn't change much. The files are slightly smaller so it may save a few writes here and there, but I wouldn't expect a drastic change one way or another.
- tmikaeld 5y agoRight, but if it's over-lapping, then wouldn't it move the write around on multiple layers?
- jbverschoor 5y agoThat's what you get when everybody is hiding abstractions in 10 layers. The same thing happens with all the layers of virtual-machines
- josteink 5y agoTo be fair you have some virtualisation stacks (like QEMU/KVM) which goes in the opposite direction: It provides VMs with drivers for purely virtual “virtio” devices (storage, network, etc) with no effort or overhead put into mimicking the mechanics of a real sas/sata/scsi device or whatever. The result is less complexity and a much more direct data-path so things should hopefully not only be more performant, but also more stable.
- deeblering4 5y agoIt's wild to me that sites managing storage clusters at petabyte scale are doing it on AWS. I would think that by then you could save millions more by migrating to your own colocated hardware.
- mwcampbell 5y agoI would hope that for a high-throughput DB cluster like this, they're using instance-local storage rather than EBS. If that's the case, then they're probably already taking advantage of EC2 reserved instances to save a lot compared to the on-demand prices that we usually see.
- jhk727 5y agoWe are, though we started out using EBS. As you mentioned, NVMe instance storage performs much better for our workload. We work around the lack of durability through strong automation of point in time restore/swapping in of new nodes in case of hardware failures. And yes, reservations make a massive difference economically.
- Stevvo 5y agoFar lower risk and capital investment than collocation. I've never had to store petabytes of data, but I would imagine the considerations are not too different to smaller scales.
- aaronblohowiak 5y agoThe amount of staff you have to have on-hand and amount of pre-planning (and up-front capital commitment) can all make that very unattractive long after the basic per-GB price would seem to make it attractive.
- b112 5y agoNo. You already have 24x7 staff at this scale. Hardware requires thought and skill, but then so does software. It isn't voodoo.
- b112 5y agoGood grief. Talk of rsync backups on live DB systems. zfs. On the fly disk encryprion. All I can think of is, clearly these guys never worked with spinning disks and large datasets. So much headroom to waste with SSDs, people are spoiled today.
- laumars 5y agoNothing you’ve described there wasn’t possible with spinning disks storing large data sets. In fact if anything, ZFS is ideally suited to exactly that scenario.
- b112 5y agoI assure you, rsync can tank a highly tuned db environment. 'updatedb', which updates the db 'locate' uses under linux, running in its cron, can cause issues. It all depends upon how much headroom you have, the type of io activity, etc. I've operated systems under consistent 80% io load with massive datasets at the time, under spinning disks. Running rsync would be madness on such a system. I know. I only did it once.
- whitepoplar 5y agoThey never said they were using rsync on their datasets. They're likely using ZFS snapshots.
- ComodoHacker 5y agoLet me guess, your system hadn't ionice back then?
- b112 5y agoionice works only on some elevators, and worked on fewer a decade ago. As well, ionice doesn't help with this degree of load.
- deleted 5y ago[deleted]
- willis936 5y agoFor anyone like me: home usage workloads are read-heavy with files that are predominantly already compressed. Moving to Zstandard might be interesting to toy with if you have more compute and disk I/O than network throughput, but the benefits would likely be smaller.
- nine_k 5y agoA semi-related story from ancient past. Back in the university days we've built an information retrieval system. It ran on an IBM PC XT, with a 20MB HDD, which was pretty slow. The heaviest queries to the information system involved full scans. They were too slow, slower than had been agreed with the customer. So we installed a disk compression program, maybe Stacker or something similar. It ate some of the already slow 8088 CPU, and some of the scarce RAM. But crucially it compressed the data to about 50% of the original size. This made the number of blocks to read, and, most importantly, to seek twice as low. The query speed increased twofold. We successfully completed the (tiny) software development contract.
- deleted 5y ago[deleted]
- bombcar 5y agoWhenever CPU speeds surpass disk speeds, compression becomes king; when the opposite happens it dies away. I don't know if we'll ever see disk speeds compete with CPU speeds again, however.
- otterley 5y ago[deleted]
- whitepoplar 5y agoThe post is literally about how ZFS compression saves them millions of dollars.
- pengaru 5y ago> The post was literally about how ZFS compression saves them millions of dollars. ... relative to their previous ZFS configuration. They didn't evaluate alternatives to ZFS, did they? They're still incurring copy-on-write FS overhead, and the compression is just helping reduce the pain there, no?
- drob 5y agoZstandard gets us 5.5x compression. The previous ZFS config got us 4.4x compression. XFS, which we ran on for years before rolling out ZFS, does not compress.
- pengaru 5y agoThanks for the clarification
- linuxandrew 5y agoThere aren't many good alternatives to ZFS if you specifically want snapshots and compression. LVM? Managing it sucks and the performance is bad (there's plenty of information on this). I think Red Hat's Stratis was supposed to improve on some of this but it's not mainlined yet. For a while it sounded like Red Hat were going down the device-mapper functionality route of Stratis, VDO, dm-integrity and XFS, but I don't know where this is at. Btrfs is probably the closest bet, but it's software RAID 5/6 has the known write hole bug. Still waiting for bcachefs to get mainlined. ext4 doesn't do CoW. XFS kinda does with its reflink functionality but it doesn't do filesystem-level snapshots.
- rsync 5y agoI am genuinely curious how a ZFS "special" device[1] which absorbs all metadata for a pool would help this kind of a workload. The "special" device is a full-blown vdev that you add to a pool (it is not a cache) - typically a 3 or 4-way mirror of SSDs. Now all metadata read and write happen at SSD speeds. We wrote about this in the rsync.net Technical Notes for Q3 of this year[2]. I know what this kind of SSD based vdev does to typical mixed file performance but I'm not sure how metadata-heavy a postgres implementation is ... [1] Yes, they really are called that. [2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_technotes.html https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...
- latk 5y agoWould separate SSD metadata devices help if the pool, as in Heap.io's case, already consists entirely of SSDs? It's obviously a win for a use case like Rsync.net's where the data is less “hot”and therefore uses much more cost-effective HDDs.
- franga2000 5y agoWould be interesting to see if Optane or even just some faster SSDs for the metadata would give any noticeable improvement. I imagine latency would be more important for metadata than throughput, so perhaps SSDs would be more or less equivalent, but I'd be really interested in seeing the numbers for Optane
- buildbot 5y agoThis is actually one of the golden use cases for optane! Back in the pre-optane day, the server company I worked for would back SSD/HDD zpools with a device called a ZeusRAM drive: https://www.ebay.com/itm/234256677232 https://www.ebay.com/itm/234256677232 It's all about the latency.
- withinboredom 5y agoThe benefits would be that metadata type FS queries are OOB from the actual data, theoretically allowing more IOPs on your data disks spent on actual data.
- djanogo 5y agoCan somebody enlighten me on why you would use ZFS for DB?, seems like there would be overlap/conflict of features. The only benefit that I know of would be quick restore to previous state, but how often would you need to restore?
- jdhawk 5y agoThey state it. Compression
- jhk727 5y agoAuthor here - It's explained in the post but the primary driver is cost. 80%+ reduction in storage is massive when you're storing petabytes of data on ssds.
- nhoughto 5y agohow important was being able to disable postgres full page writes under zfs? that was almost a throw-away line in the article but could be really important for some pg workloads i'd think? certainly got my attention
- jhk727 5y agoIt was a nontrivial improvement to write throughput at the time. I'd imagine it could have similar impact for other write-intensive workloads.
- 5y ago
- mikewarot 5y agoMy understanding of SSD architecture is that you can't flip bits in a page, you have to write a page at a time, thus all SSD systems stall if they get stuck waiting for empty pages (which take longer than writing pages). Thus, a full SSD (which internally has a few % allocated spares the customer isn't supposed to be able to access) is a slow SSD. It would seem to me if you can keep the utilization of the disk under 80%, and support TRIM (which lets the SSD know which pages can be erased), you should be able to get really high performance out of them with a Copy on Write file system.
- Neil44 5y agoI was about to post the same, you will quickly run out of ready trim’d blocks at high utilisation, which the article doesn’t mention.
- nwmcsween 5y agoOK a one thing that stands out here and please correct me if I'm wrong: > ... multi-petabyte cluster of Postgres instances... blocksize relatively high at 64 kb ... The dataset should be the Postgresql "page size" which IIRC is 8KB, the reasoning for this is RMW cycles will read 64kb modify 8KB and write out the full 64KB amplifying writes 8 fold. Also IIRC Postgresql will automatically use TOAST when needed?
- jhk727 5y agoGood callout - we use a higher blocksize than Postgres page size because it gives us a much higher compression ratio, at the cost of some read/write amplification. And yes - Postgres will automatically TOAST oversized tuples and compress the relevant data (if you configure it to do so). This is much lower impact for us than filesystem level compression, as it doesn't affect the main relation heap space (or any indexes).
- nwmcsween 5y agoThere has to be something better than a potential 8 fold write performance reduction wrt compression
- nwmcsween 5y agoWhat about: https://people.freebsd.org/~seanc/postgresql/scale15x-2017-postgresql_zfs_best_practices.pdf https://people.freebsd.org/~seanc/postgresql/scale15x-2017-p... 16k record size 2x amplification and still (?) allows compression w/ lz4
- jhk727 5y agoWe tested this extensively a few years back. We saw a compression ratio of ~1.9 with 8k recordsize/lz4, ~2.7 with 16k/lz4, and now ~5.5 with 64k/zstd.
- infogulch 5y agoThis is pretty interesting, because the effects of migrating from lz4 to zstd were: - Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes) - Average write operation duration decreased by 50% on our fullest machines - No observable query performance effects It seems like the better compression ratio and resulting reduced IO more than makes up for increased CPU compared to lz4. I wish they had mentioned the actual effect on CPU. Compare to the recent thread "The LZ4 introduced in PostgreSQL 14 provides faster compression" [0] where the loudest voices were saying that zstd would not work due to increased CPU. This is a different layer (filesystem compression vs db compression), but this article represents an interesting data point in the conversation. [0]: https://news.ycombinator.com/item?id=29147656 https://news.ycombinator.com/item?id=29147656
- walrus01 5y agoI wonder what, if any, further improvement would be had by comparing xzip vs zstd. Obviously you need a LOT of CPU to throw at xzip if you want to use it. zstd is very much more optimized for compression at speeds comparable to traditional gzip. I use xzip primarily for things that will get compressed to long term storage and the time to create the archive isn't a really important factor. in this test: https://sysdfree.wordpress.com/2020/01/04/293/ https://sysdfree.wordpress.com/2020/01/04/293/ zstd level 19 wins on time vs. xz levels 5 through 9, but the xz ultimate compressed file size is definitely smaller.
- ncmncm 5y agoIf your system experiences periods of greater and lesser load, then using the rest of whatever is its load capacity, during periods of lesser load, on further compressing its contents might be worth the bother. Perhaps better than stepping to a different compression algorithm, zstd has multiple levels of compression that might be used at different times. The advantage there is that the same decompression algorithm works for all. One might reasonably hope that decompression tables may be shared amongst multiple of the 64k raw blocks, to further squeeze usage.
- mayli 5y ago
- peremasip 5y agoThis is pretty interesting, because the effects of migrating from lz4 to zstd were: - Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes) - Average write operation duration decreased by 50% on our fullest machines - No observable query performance effects It seems like the better compression ratio and resulting reduced IO more than makes up for increased CPU compared to lz4. I wish they had mentioned the actual effect on CPU. Compare to the recent thread "The LZ4 introduced in PostgreSQL 14 provides faster compression" [0] where the loudest voices were saying that zstd would not work due to increased CPU. This is a different layer (filesystem compression vs db comrpession), but this article represents an interesting data point in the conversation. [0]: https://news.ycombinator.com/item?id=29147656 https://news.ycombinator.com/item?id=29147656
- magicalhippo 5y ago> For these reasons, it’s generally recommended not to let ZFS go past 80% disk utilization. There's another reason why you don't want to go beyond 80% utilization, and that's because the block allocator will switch behavior to a more involved search, which can take a lot more time. Thus allocating new blocks can get really slow once you get past 80%.
- k8sToGo 5y agoDoes the problem go away again if you go back to below 80%?
- magicalhippo 5y agoAfter a bit of digging, yes but no. So turns out it's a bit more involved than what's been commonly told as a straight up 80% == bad scenario. ZFS by default divides[1] each vdev (RAIDZ or mirror set) into ~200 allocation regions called metaslabs[2]. When allocating from a metaslab[3] it will check if the free space in that metaslab is below the threshold defined by metaslab_df_free_pct. It seems the threshold was changed to 4% free space at some point[4]. If the free space is above the limit it will use the fast first-fit search, if not it will use the expensive best-fit search. However, as noted that threshold is per metaslab. So if the pool is fragmented, even though the overall free space in the pool is above the 4% threshold, there might be metaslabs with less than that free, which will lead to the expensive best-fit search. So it's not a hard limit, but it should start to be noticeable above 80%. [1]: https://www.delphix.com/blog/delphix-engineering/openzfs-code-walk-metaslabs-and-space-maps https://www.delphix.com/blog/delphix-engineering/openzfs-cod... [2]: http://dtrace.org/blogs/ahl/2012/11/08/zfs-trivia-metaslabs/ http://dtrace.org/blogs/ahl/2012/11/08/zfs-trivia-metaslabs/ [3]: https://github.com/openzfs/zfs/blob/master/module/zfs/metaslab.c#L1666 https://github.com/openzfs/zfs/blob/master/module/zfs/metasl... (note metaslab_df_free_pct) [4]: https://www.truenas.com/community/threads/zfs-tweak-for-first-fit-versus-best-fit.11817/ https://www.truenas.com/community/threads/zfs-tweak-for-firs...
- linuxandrew 5y agoAs you mention, it differs from system to system. When it hits that threshold it becomes immediately noticeable - I/O performance will basically fall right down. At my old job we noticed it around 93% on Solaris 11.x, and maybe 85% on Solaris 10 off the top of my head, on two different production systems. Best practice was considered to keep pools on any ZFS system (OpenZFS or Solaris) below 80%.
- whitepoplar 5y agoKinda unrelated, but I have a question for anyone who is knowledgeable about running Postgres on ZFS...does setting a large-ish ZFS block size (e.g. 64kB) for use with Postgres (default 8kB blocks) cause a great deal of write-amplification even when ZFS `full_page_writes = off`?
- B1FF_PSUVM 5y agoI don't have a dog in that race, but I've seen it said that the DB architecture itself should be reviewed, because SSDs make it possible to use databases in higher "normal form", with more tables that require more lookups, but less data volume. E.g. https://drcoddwasright.blogspot.com https://drcoddwasright.blogspot.com: "In a time of SSD, multi-core/processor, two terabyte memory and Optane App Direct Mode machines, there is no reason not to build from BCNF data. Time to do what Dr. Codd demonstrated. Technology has finally caught up with the maths."
- nhoughto 5y agoLots of interesting nuggets in here, i'm also interested in the ASG driven db node replacement approach, has anyone seen any more info about the nuts and bolts of how this works? The idea of treating db nodes less like pets seems great, there must be a few tricks to it tho..
- rob_c 5y agoTo summarise _UPGRADE YOUR ZPOOL_ and _UPGRADE YOUR ZFS MODULE_. Best part of ZFS IMO is that the on disk data is well formatted and has integrity checking built in as standard. Old adage, but. If the data cost money to produce or is valuable invest in ECC and on-disk integrity checking. Managing a few Pb I'll also emphasise updating your kernel to mainline of possible, especially if your on a centos like distro. These 2 things alone can potentially net huge performance gains.
- almostdigital 5y agoIn case someone else is wondering why this company stores petabytes of data, this is why: > Heap is the only tool that automatically captures all user interactions on your site, from the moment of installation forward. A single snippet grabs every click, swipe, tap, pageview, and fill — forever. Revolting... at least my content blocker had their collection script already blocked.
- tgtweak 5y agoI mean, it's first party data (stored on behalf of the site you're visiting) accessible to that first party vs data that is going to Google Analytics or Facebook (3rd party) and which is accessible by all of Google or Facebook (actually concerning). Not trying to make a case for large scale data collection without purpose - but if clickstreams are going somewhere I would rather it no be to a company selling me ads.
- lrem 5y agoOn the other hand, Google has pretty strict data retention rules. For Analytics: > The maximum amount of time that Analytics will retain Google-signals data is 26 months, regardless of your settings. First party can be longer, but is usually shorter than this. Disclosure: I'm a Googler. Logs of my product are retained for 14 months and until recently it was 1 month, but it turned out we need to debug cases older than that.
- almostdigital 5y agoYou'd rather it be a company that sells your data to as many third parties as possible? Because that's what they all do. Google and Facebook are arguably better because they at least are somewhat incentivized to keep your data to themselves. Even though that's like arguing about which STD is better instead of using condoms.
- anaganisk 5y agoDidn’t Cambridge Analytica happen exactly due to this? Data “leak” to third parties?
- gigatexal 5y agoTangential question here: has heap.io done any articles on ZFS and DBs with respect to any tax that CoW nature of ZFS has on DB txns/sec?
- kriro 5y agoHow would one go about benchmarking file systems for a new server with no extra machines at hand? I've ordered one for our research lab and it's not mission critical in any way and we have our sweet time to set it up. Is there anything more advanced than: default bare metal install with FS of choice, run benchmark, reinstall with different FS, rinse repeat? Or do you test with VMs?