4 ms·
What sorts of programmers should be concerned about these matters? Page cache doesn't seem too important or interesting in my day to day app and distributed sy
by metadat 4y ago
What sorts of programmers should be concerned about these matters? Page cache doesn't seem too important or interesting in my day to day app and distributed systems development.
Maybe it's useful if you want to make something like a more performant version of grep? (aka ripgrep?)
- Gordonjcp 4y agoBy the looks of the article? People writing SSD firmware, or SSD drivers. There is probably a small but non-zero number of these on here.
- jqcoffey 4y agoThe author appears to be an EM at Booking.com. It seems unlikely that anyone at Booking would be working on SSD firmware or drivers, but a CDN seems like a reasonable assumption and also a useful place to plumb the depths of SSD implementations.
- eschneider 4y agoPeople who read from disks and people who write to them. How SSDs organize data definitely had read and write performance implications and if you're writing to disk, some write habits that are perfectly reasonable on regular disks can cause catastrophically fast wear on SSDs.
- KennyBlanken 4y agoYes, but the number of people who need to be worried about aligning their writes and such is pretty small; certainly not "every" programmer. The author gets into the weeds about certain things application level programmers almost never need to know or concern themselves about. He really doesn't understand what's useful information and what isn't. If you're programming at enterprise scale, this sort of stuff is the responsibility of architect-level programmers and senior systems engineers. Even most linux sysadmins know all about block alignment (well, if they predate most of the various tools figuring out block size/alignment stuff for you.) It's nothing new - RAID arrays work best when properly aligned, for example.
- loxias 4y ago> Page cache doesn't seem too important or interesting in my day to day app and distributed systems development This is why we can't have nice things.
- the_only_law 4y agoI love how people say this, when the reality is, all the software from the oh-so-coveted is the biggest shit show I’ve seen. But it’s rarely because some developer didn’t understand page caches, and usually because it obviously didn’t revive enough QA or UX input.
- tshaddox 4y agoHow so? Isn't the only point of developing these systems and abstractions so that other people don't have to worry about them?
- chrisandchris 4y agoIMHO, today to many people think "don't have to worry about them" equals "don't need to know anything about it".
- tshaddox 4y agoI would argue that in most cases you "don't need to know anything about it" either. It's reasonable to deliberately treat abstractions as if they are not leaky, as long as you're aware that all abstractions in fact are leaky and you're equipped to investigate and learn about them if the leaks cause problems.
- dotopotoro 4y ago“don't need to know anything about it” is acceptable, but should not be encouraged. It’s not like reading 10 bullet points on the subject is “diving deep” and making huge time investment. It’s just getting the minimal context, so later on at least some keywords are known.
- yourapostasy 4y agoNot just programmers. Anyone using ZFS with SSD, whether as the pool itself or in various caches like slog(zil) is going to find this information of use when tuning for better SSD citizenship. Programmers treating SSD like faster spinning rust is like programmers treating S3 like another POSIX filesystem; you can do it, but you're trading away compounding future advantages for that one moment of expedience.
- dekhn 4y agoIn my career I have found that file system tuning for the devices an anti-pattern that almost always ends up causing more problems than it's worth.
- philjohn 4y agoAre you writing low-level software, such as filesystems, or raw block backed database storage engines? If not, then that's definitely a decent maxim to live by.
- golergka 4y agoDon't your distributed systems use databases of some sort?
- alpaca128 4y agoAnd why does a DB user need to know those details? Isn't it the whole point of DB systems to provide an optimized solution that allows users to focus on other things?
- dotopotoro 4y agoDatabases always try to flush something to disk after transaction, just in case unexpected reboot happens. So your writes to db have direct correlation to disk writes. Choice of db schema impacts physical layout on ssd. E.g. Different tables are more likely to be on different ssd pages resulting in random writes. Databases are insanely complex, but not magic.
- jeffbee 4y ago> doesn't seem too important or interesting in my day to day app and distributed systems development. Makes sense to me. At Google we were told to stop thinking about all this stuff, that the storage hardware and software people were responsible for hiding things like wearout from application developers. This article is really "things you should know if you plan to directly access an NVMe device" but there is a huge class of programmers who are better off not knowing.
- bombcar 4y agoThere was an article by varnish taking about how you should leave the caching and memory management to the OS - even if you can beat the virtual memory manager today you’ll stop improving your home grown solution while RAM and the kernel keep marching on.
- wmf 4y agohttps://varnish-cache.org/docs/trunk/phk/notes.html https://varnish-cache.org/docs/trunk/phk/notes.html
- rasz 4y ago>At Google we were told to stop thinking about all this and as a result Chrome slams SSD by writing cached Youtube videos to disk .... except Youtube never reuses cached video data (not even when rewinding more than couple minutes to already watched spot in same video), it explicitly generates hashed requests with custom URL parameters googlevideo.com/videoplayback?expire (~6hour shelf life) &range &sig &lsig. Heavy YT viewing results in wearing out your SSD by tens of gigabytes per day for no particular reason. This is just one small example of side effects from such brilliant decisions.
- MichaelZuo 4y agoYikes, is that really the case? No wonder the 128GB drive on my 2013 MacBook air wore out to 60% of its original performance...
- pavon 4y agoMy take: 1-13) General background info that informs the rest. 14-25) Important for any programmer that does enough file IO that they need to optimize it. 26-29) Important for any system admin to ensure they aren't inadvertently limiting the performance of their hardware.