10 ms·
I suspect the biggest (build time) benefit to most c++ workflows and toolchains was the move to ubiquitous SSD. Prior to that in my experience excepting expen
by ska 6y ago
I suspect the biggest (build time) benefit to most c++ workflows and toolchains was the move to ubiquitous SSD. Prior to that in my experience excepting expensive RAID array dedicated build machines, it was really easy to build a system that would always be IO bound on builds. There of course were tricks to improve things but you still tended to hit that wall unless your CPUs were really under spec.
edit: to be clearer, I'm not thinking of dedicated build machines here (hence RAID comment) but over all impact on dev time by getting local builds a lot faster.
- trhway 6y agoi (and some teamates) actually put HDDs on some workstations as SSD just die after 2-3 years of active build on them and with modern HDDs you have practically unlimited storage while you can have only limited number of 400G builds on SSD (the org has psychological barriers to having more than 1-2Tb SSD in a machine) and the SSD start to have perf issues when at 70-80% capacity . With HDD the build time didn't change much - the machines have enough memory for the system to cache a lot (256-512G RAM).
- outworlder 6y ago> i actually put HDDs on some workstations as SSD just die after 2-3 years of active build on them That sounds very low for modern SSDs, even consumer-grade. Have you tried different vendors?
- winkeltripel 6y agoIf spending hours a day at 100% utility, SSDs will rarely last 5 years.
- kyrra 6y agoDepends on the kind of SSD. If it's using SLC, the write endurance is much much higher. If you're going with cheap SSDs (TLC or QLC), your write endurance will suck. see: https://www.anandtech.com/show/6459/samsung-ssd-840-testing-the-endurance-of-tlc-nand https://www.anandtech.com/show/6459/samsung-ssd-840-testing-...
- mywittyname 6y agoSLC seems to be going away pretty quickly, if it hasn't already been phased out. It just can't produce the price / GB of the better tech. Also, that article you linked is almost 10 years old. You're best bet for long-term reliability is to buy much more capacity than you need and try not to exceed >50% capacity for high write frequency situations. I keep an empty drive around to use as a temp directory for compiling, logging, temp files, etc. Also, my understanding is that consumer-grade drive need a "cool down" period to allow them to perform wear leveling. So you don't want to be writing to these drives constantly.
- pdimitar 6y agoI recently bought an external 32GB SLC SSD (in the form factor of an USB pendrive). Its random read/write speeds are quite insane (130+ MB/s both) while consumer SSDs like the Samsung 850 Evo barely manage 30 MB/s read/write. It's also advertised as very durable. I plan on using a couple of those as ZFS metadata and small block caches for my home NAS, we'll see how it goes but people generally and universally praise the SLC SSDs for their durability. > You're best bet for long-term reliability is to buy much more capacity than you need and try not to exceed >50% capacity for high write frequency situations. I keep an empty drive around to use as a temp directory for compiling, logging, temp files, etc. That's likely true. I am pondering buying one external NVMe SSD for that purpose exactly.
- 1996 6y ago> I recently bought an external 32GB SLC SSD (in the form factor of an USB pendrive). Its random read/write speeds are quite insane (130+ MB/s both) Which brand/spec?
- pdimitar 6y agohttps://www.aliexpress.com/item/32846479966.html?spm=a2g0s.9042311.0.0.6a514c4doNgGGl https://www.aliexpress.com/item/32846479966.html?spm=a2g0s.9... Vtran / eVtran. They have favourable reviews, too.
- read_if_gay_ 6y agoWhat kind of workload will do that?
- leetcrew 6y agoa build server recompiling multiple branches over and over in response to changes.
- dragontamer 6y agoAnd logging all of those unit tests associated with all of those builds (and rolling over those logs with TRACE level debugging). Every build gets fully tested at maximum TRACE logging, so that anyone who looks at the build / test later can search the logs for the bug. 8TBs of storage is a $200 hard drive. Fill it up, save everything. Buy 5 hard drives, copy data across them redundantly with ZFS and stuff. 1TBs of SSD storage is $100 (for 4GBps) to $200 (for 7GBps). Copy data from the hard drive array on the build server to the local workstation as you try to debug what went wrong in some unit test.
- officeplant 6y agoMy mind was blown when I had to send a fully logged 5 minutes of system operation to a friend for diagnostics (MacOS 11 on an M1 Mini). He wasn't joking when he said don't go over a few minutes because the main 256GB system drive almost ran out of space in that time. After getting it compressed down from 80GB and sent over I got my mind blown again when he explained he had to move to his workstation with 512+gb of ram just to open the damn file.
- Retric 6y agoIf your SSD would be at 100% utilization it’s going to take a lot of HDD to reach that kind of bandwidth. To the point where for high bandwidth loads SSD’s actually cost less even if you have to replace them regularly. 100% utilization and 30x the bandwidth = 30x as many HDD. Alternatively, if HDD’s are an option you’re a long way from 100% utilization.
- trhway 6y agoSSD have hard time sustaining 200-400Mb/s write where is 4 HDD do is easily. Our case isn't that much about IOPS. Anyway, reasonably available SSDs have up to [1000 x SSD size] total write limit, so doing couple of 400G builds/day would use up the 1TB drive in 3 years. At worst times we had to develop&maintain 5 releases in parallel instead of regular 2-3.
- deleted 6y ago[deleted]
- Sesse__ 6y agoUrm what? A modern NVMe drive will sustain ~2 GB/sec write. (See e.g. https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAXk6do4SSf8-1280-80.png https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAXk6do4SSf8-128..., from Tom's Hardware)
- adrian_b 6y agoFew SSDs can sustain such a speed for a long time. After they exhaust their temporary SLC cache they drop to much lower speeds. SSDs that have accumulated a large amount of writes might also make large pauses at random times during writing, for garbage collection. Good SSDs remain faster than HDDs even in these circumstances, but they are nevertheless much slower than when benchmarked for short times while they are new.
- Retric 6y agoYou can get SSDs without such issues. At least for the kind of long sequential writes followed by deletes where HDD are are even vaguely comparable, good SSD’s can maintain very close to their theoretical maximum speeds indefinitely. They only really slow down in usage patterns that trash traditional HDD’s, which have much worse disk fragmentation issues.
- ska 6y agoThose are pretty beefy workstations, does every developer have one or are these really build servers? As I noted you could always throw money at this and end up somewhere reasonable, but it introduces workflow considerations for your devs.
- EricE 6y agoA way to have your cake and eat it too - check out Primocache. It's pretty inexpensive disk caching software (especially for Windows Server which is where I really leverage it!). Pair it with an Optane for L2 cache and it will speed up normal SSD use too ;)
- fctorial 6y agoHow does it compare to default linux caching algorithm?
- Dylan16807 6y agoThe default one that caches disk data in memory? It solves a very different problem by caching data from slower disks onto faster disks. It can accelerate reads and it can also act as a writeback cache.
- Filligree 6y agoIt's most comparable to lvmcache or bcache. As in, Primocache does three actually useful things: - It can pre-populate an in-memory disk cache. Not massively useful, but depending on your workload and uptime it might save some time. Nothing I know of does this on Linux. - It can act as a level 2 block cache, i.e. caching frequently accessed blocks from slow (HDD) storage on fast (SSD) storage. This is massively useful, especially for e.g. a Steam library. - It can force extensive write buffering, using either main memory or SSD. And yes, it can also act as a block buffer in main memory, but I don't find that helpful. Windows' default file cache does a good enough job, and Linux' VFS cache works even better. (Though ZFS' ARC works better yet...) Its write bufferin increases write speeds massively, inasmuch as it delays actually writing out data. Obviously, doing so with main memory means a crash will lose data; what's not so obvious (but amply explained in the UI) is that, because it's a block-based write buffer, it can also corrupt your filesystem. Primocache does not obey write barriers in this mode. What's even less obvious, and where it differs from lvmcache / bcache, is that this still happens if you use only an L2 write buffer, not a memory-based one. The data will still be there, on the SSD, but Primocache apparently doesn't restore the write-buffer state on bootup. Possibly it's not even journalled correctly; I don't know, I just got the impression from forums that fixing this would be difficult. So, overall, bcache / lvmcache / ZFS* are all massively superior. Primocache is your only option on Windows, however, unless you'd like to setup iSCSI or something. I've considered that, but I'd need at least a 10G link to get worthwhile performance. *: ZFS supports cache-on-SSD, and unlike a year ago that's persisted across reboots, but it doesn't support write-buffer-on-ssd. Except in the form of the SIL, which is of dubious usefulness; that buffer is only used for synchronous writes. However, ZFS is a transaction-oriented filesystem. It never reorders writes between transaction groups, which means that if you run it with sync=disabled -- which disables fsync & related calls -- it's still impossible to get locally visible corruption. The state of your filesystem at bootup will be some* valid POSIX state, as if the machine was suddenly but cleanly shut down. You still need to tweak the filesystem parameters; by default transaction groups end after 5 seconds, which is unlikely to be optimal. Alternately, you can run it on top of lvmcache or bcache.
- fctorial 6y agoWhat are the signs of an SSD that's about to die?
- trhway 6y agoanecdotally i'd see filesystem errors and write speed dropping, sometimes writes practically hanging, especially if the drive is close to 90% capacity.
- zlynx 6y agoSMART monitoring. In my experience the SMART data on SSDs is pretty good at pointing to failures. Either that or the SSD simply disappears from the system. But that happens to HDDs also.
- PragmaticPulp 6y agoSSDs help, but nothing beats core count X clock speed when compiling. Source code files are relatively small and modern OSes are very good at caching. I ran out of SSD on my build server a while ago and had to use a mechanical HDD. To my surprise, it didn’t impact build times as much as I thought it would.
- ska 6y agoHmm. Maybe the tradeoff has changed since I last tested this (to be fair, a few years ago). But I'm also not focused on build servers especially, it's always been possible to make those reasonably fast. Unless you have a very specific sort of workflow anyway, your devs are doing way more local builds than on the server and that sped up a ton moving to SSD, in my experience anyway. YMMV of course.
- fctorial 6y agoDon't SSDs have a finite TBW? 50GB of writes everyday (possible on large projects) will consume that in a couple of months.
- frizkie 6y agoThey do but it's really large. The Tech Report did an endurance test on SSDs 5-6 years ago [0]. The tests took 18 months before all 6 SSDs were dead. Generally you're looking at hundreds of terabytes, if not more than a petabyte in total write capacity before the drive is unusable. This is for older drives (~6 years old as I said), and I don't know enough about storage technology and where it's come since then to say, but I imagine things probably have not gotten worse. [0]: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/ https://techreport.com/review/27909/the-ssd-endurance-experi...
- yread 6y ago> things probably have not gotten worse. I am afraid they did, consumer SSDs moved from MLC (2 bits per cell) to TLC (3) or QLC (4). Durability changed from multiple petabytes to low hundreds of terabytes. Still a lot, but I suspect the test would be a lot shorter now.
- Dylan16807 6y agoThe TLC drive in that test did fine. 3D flash cells are generally big enough for TLC to not be a problem. I would only worry about write limits on QLC right now, and even then .05 drive writes per day is well under warranty limits so I'd expect it to be fine. Short-lived files are all going to the SLC cache anyway.
- dekhn 6y agoI've swapped hundreds of terabytes to a terabyte SSD (off the shelf cheapie) with no recognizable problems (the gigapixel panoramas look fine).
- 6y ago
- lumost 6y agoYou know I wonder how much of an impact this has had on the recent move back to statically typed and compiled languages vs. interpreted languages. I had assumed most of the compilation speedups were due to enhancements to the compiler toolchain - but my local laptop moving from 100 IOPS to > 100k IOPS and 3GB/s throughput may have more to do with it.
- rbanffy 6y agoCPUs got faster too. My MacBook pro is a lot faster than the 6yo top of the line Mac Mini. In fact, if compile times are being limited by storage there should be some quick wins in configuration terms - building intermediates to RAM, cache warming, etc - that can enable better performance than faster storage.
- KingOfCoders 6y agoWe had ultra fast HDDs as developers with sound proof housings because they were so loud. Glad for SSDs.