10 ms·
Swap on HDD: Does placement matter?
- pbhjpbhj 5y agoNice write up. I ditched swap partitions a few years ago, my system (home computer) basically never swapped. At the time I was digging myself out of a too-small /boot (a once-recommended size) and figured that one-big-partition with a swap-file gave most flexibility. So, is the an efficient way to leverage the speed improvement for other than swap -- like binary caching of executables of some form?
- szszrk 5y ago> So, is the an efficient way to leverage the speed improvement for other than swap -- like binary caching of executables of some form? Sure. Like the site I mentioned in other comment, more or less "out tracks are faster". But this applies just to HDD drives. It's mostly useless for modern infrastructure, like SAN (even HDD based), all kinds of SSD and so on. A curiosity - last time I saw a cool optimization of HDD usage was on "old gen" consoles, like ps4 and xbox one. Most games duplicated assets multiple times. Games took much more GB then needed, but the drive did not had to jump between many HDD tracks so much and it mattered for instance in big open world games.
- pizza234 5y agoThe system uses swap to put less frequently used pages; the freed RAM can be used for caching, or more in general, pages that are used more frequently. So adding swap indirectly increases the memory available for caching. I don't know the details of (Linux) caching, though. On my (32 GB) system, there are a few completely unused GB, it seems.
- deleted 5y ago[deleted]
- kijin 5y agoThe most surprising thing about the result is that there isn't an order-of-magnitude jump between SATA SSD and any sort of HDD, as you would expect with random read/write workloads typical of swap thrashing. Instead, the chart looks as if it is mostly measuring sequential read/write performance. HDDs have long been known to be faster on one end than the other in sequential benchmarks. This could be an artifact of the particular kind of workload that the author used. Maybe it causes large numbers of adjacent blocks to be swapped in and out at the same time?
- koala_man 5y agoAuthor here. In all cases, most access is still RAM. The storage is only hit to stash or load overflowing pages. I originally ran the benchmark with 1GB RAM instead of the final 2GB, but the start-of-disk test did not finish in the 9 hours I let it run. With 0GB, I don't doubt that you'd see the expected 1,000,000x latency difference between disk and DRAM.
- deleted 5y ago[deleted]
- szszrk 5y agoI believe this was kind of obvious 15-25 years ago [1] . That was in THE basic tutorial [2]. Those were simpler days. It was hard to build something big by yourself. Now it's easier, but now we are learning just API's to API's that provision our hardware and software :) So much current dev and ops knowledge will be useless in a few years, yet I could easily use a book from 1970' that was recommended here one day, to learn and use some basic AWK nowadays. [1] Example from 2007 https://www.linuxquestions.org/questions/debian-26/debian-install-swap-partition-516216/#post2571506 https://www.linuxquestions.org/questions/debian-26/debian-in... [2] Example from around 1997 https://tldp.org/HOWTO/html_single/Partition/#SwapSize https://tldp.org/HOWTO/html_single/Partition/#SwapSize
- iforgotpassword 5y agoYes, this was common knowledge, yet back in the day most distro setups still put swap at the end by default for reasons unknown to me. Apart from the speed issue, that also made moving an existing installation to a larger disk more complicated, since you couldn't just resize the os partition, you had to delete and then recreate swap.
- minitoar 5y agoI thought there was some idea that you wanted core os/app data near the center since you would always be using that.
- ragnese 5y agoYes, that was the argument I remember reading. You put "system" stuff first, then /home if you did a separate partition for it, etc. Swap last because "hopefully you won't be swapping much anyway". I also (vaguely) remember some people putting build partitions closer to the front.
- actually_a_dog 5y agoI think we're also talking about the days when a machine that was swapping extensively was going to be stupidly slow no matter what you did.
- tomxor 5y agoThis is due to platter geometry... the "start" of the logical volume is at the outer edge of the platters, and the end is at the inner edge. If you divide the platter into concentric circles of equal width, you will notice there is more area available on the outer circles... for this reason the number of sectors per track are greater the further the track is from the centre of the platter. Yet the head will pass over the entire track in the same amount of time... i.e more data in the same time. It makes sense that the logical volume would be arranged from the outer edge to take advantage of the speed as soon as possible.
- trhway 5y ago> for this reason the number of sectors per track are greater the further the track is from the centre of the platter. Yet the head will pass over the entire track in the same amount of time... an additional consequence is that for the same amount of data it takes lesser number of tracks thus making for faster/shorter seeks inside that data.
- fomine3 5y ago> the "start" of the logical volume is at the outer edge of the platters, Is this a something like natural theory, or de facto standard? Maybe the start can be most inner edge on HDD in another planet?
- tomxor 5y agoIt's pretty universal, as inevitable as pythagoras theorem, even on an alien planet where everything else arbitrary and historical about the technology is different: the geometry dictates this optimisation will eventually be made (zone-bit-recording) [0]. Once the optimisation is made, it's also inevitable that logical layout will start from the outer cyclinder to provide the best sequential performance at the beginning of it's use. Older HDDs did not make this optimization, they had a constant number of sectors per cylinder (track), and they also didn't come with HDD controllers, or came with more minimal controllers, and much like old floppy drives they exposed a lot of the physical layout and properties to the host system. This is why the CHS format is a historical part of OS and partitioning, modern HDDs only expose LBA. Anyway, my point is that with the older drives that lacked ZBR there is no obvious "natural law" dictating that you shouldn't create a logical layout from the inner edge - and since they exposed CHS to the OS I wonder if it was possible to chose the layout direction. [0] https://en.wikipedia.org/wiki/Zone_bit_recording https://en.wikipedia.org/wiki/Zone_bit_recording [edit] I'm trying way too hard to entertain your idea, but now I thought it I gotta write it: the one way that ZBR could exist at the same time as it making sense to start the logical volume at the inner track (or more correctly making no difference), is if the angular velocity was not constant... modern HDDs have a constant angular velocity (constant RPM), and assuming the head's maximum read capability is the linear speed at the outer cylinder, then in theory the drive could spin faster when the head is closer to the center to provide the same linear speed, at which point the sequential performance is almost constant (cylinders sector density is segmented as the area passes a threshold, so there will be a subtle periodic difference). This is how CDs work (ignoring the fact that they are a spiral), they have a constant linear speed rather than a constant angular speed and they start from the center, so the motor has to continuously change RPM to maintain it. I suspect the reason this is not used is that HDDs spin much faster than CDs, and rather than single spiral track there are cylinders that the head has to move between, the head can move between cylinders much faster than the drive motor can accurately make the subtle changes required to achieve a constant linear speed. The end result is probably very poor seek performance (i.e the seek performance would be limited by the motor rather than the head - which is very fast)... I don't know much about CDs but I suspect they also have poor seek performance, but by choice - that may have more to do with being one giant spiral track rather than motor performance.
- marcodiego 5y agoInteresting idea: multiple swap partitions... the kernel smartly chooses the closer one the write head whenever needed.
- timvdalen 5y agoCouldn't that result in much slower reads when the head is far away from the swap when it needs to read (multiple times)?
- marcodiego 5y agoEven more interesting idea: pages are oportunistically mirrored between swap partitions and the kernel smartly chooses the closest one whenever needed!
- resonator 5y agoDoes the kernel even have the information to know which is closest. I figured that would be absracted away to the disk controller.
- marcodiego 5y agoThe slowest swap was at the end probably because it was farther. The position of the head can be inferred by the geometry and last access.
- teddyh 5y agoIIRC, you can already set priorities on different swap partitions, so that the kernel chooses the ones you want it to use first.
- marcodiego 5y agoYes,but it only changes the swap partition/device/file once another one is full.
- 5y ago
- birdman3131 5y agoThe term you are looking for is "Short Stroking" and has been around for a long time. Before SSD's got cheap enough it was occasionally used where it was worth the cost of only using 25% or less of the drives capacity.
- raffraffraff 5y agoWith spinning rust the ideal usage pattern is sequential: like writing a large file from the start of a disk into contiguous sectors (or reading that file back). One of the things that screws up HDD performance much worse than placement of files on disk is randomness in the usage pattern. The mechanical nature of a HDD means that when you read and write lots of small files in different sectors, the head spends more time moving around than reading or writing. Back when we used to defragment Windows filesystems, we doing a bunch of up-front disk optimization to organise files into continuous chunks so they could be read back quickly when needed. The biggest problem I have seen with these situations is that you don't have direct control over the order of operations that the disk will be asked to perform. You think that because your file is written contiguously that it will be read that way. But depending on how busy the system is, that might not be the case. Where many processes are contending for disk access, and especially when the kernel is doing a lot of swapping to the same device, that head might be racing back and forth regardless of your file placement, and your disk performance goes straight into the toilet.
- lloydatkinson 5y agoYou still do defragment HDDs today
- zozbot234 5y agoModern file systems do not need defragmenting. It was something that was only really done with FAT.
- adancalderon 5y agoI am an old time slackware user and I always put swap at the beginning on 3.5 inch drives. I belive for laptop drives the end of drive was better.
- bluedino 5y ago165s (2:45) — RAM only 451s (7:31) — NVMe SSD Good argument for when the uninformed state that "NVMe might as well be RAM"
- californical 5y agoI mean that’s really close. I always thought of RAM as multiple orders of magnitude faster than disk. Within 3x of speed is pretty excellent. (though, I guess this doesn’t give us any latency info, just throughput. I’d expect RAM latency to still be faster)
- nh2 5y agoI would not take this benchmark to draw general conclusions. The spinning disk result is only 10x slower than RAM. But a spinning disk's throughput is 100-1000x less than current RAM, and for latency it's even worse. Similarly, the other factors in the benchmark graph are way off their hardware factors. This benchmark is measuring how one specific program (the Haskell Compiler compiling ShellCheck) scales with faster memory, and the answer is "not very well".
- koala_man 5y agoThe overwhelming majority of access would still happen in the 2GB RAM the benchmark has. The disk is only hit to stash or load overflowing pages, not on every memory access. That's why it doesn't mirror the hardware difference between DRAM and disk.
- nh2 5y agoThat makes sense, thanks!
- koala_man 5y agoAuthor here. Keep in mind that most access in the swapping case is still RAM, so we can't just say that there's a 3x difference between DRAM and NVMe flash. I originally tried running the test with only 1GB RAM, but killed the job after 9 hours of churning.
- drewg123 5y agoIn traditional BSD unixes, swap was always the second partition, very close to the front of the disk, and just behind a small root/boot partition (which was first, presumably due to bootstrapping needs). Looks like the old timers knew what they were doing :)
- kijin 5y agoI still do that with my Linux boxes. A small /boot goes first, followed by swap, /, and finally the partition that will contain the majority of the data (usually /var or /home). This arrangement has advantages even in the age of VMs and SSDs. If I want to change to a larger disk or array (or resize the virtual disk), I can simply extend the last partition where the extra space is most likely to be needed. If swap was last, it would get in the way. On the other hand, if I needed more swap, I could just add a swapfile somewhere.
- mkl95 5y agoI've always assumed this to be the proper way to do things, and it's still what I do when I'm asked to configure a new VM. If it ain't broke, don't fix it.
- 5e92cb50239222b 5y agoswapoff + fdisk + mkswap + swapon takes two minutes tops. I much prefer that to partition "fragmentation".
- chousuke 5y agoI have the UEFI/boot partition followed by an LVM PV. If it's a VM, data disks just get the whole disk, though I still usually set up LVM because it enables stuff like live storage migrations. I've actually had to do those more than once in production; one migration involved moving several terabytes of data used by a hardware server from an aging SAN onto physical disks and iSCSI. It required no downtime. I haven't really had to worry about partitioning on any Linux machine I manage for over a decade thanks to LVM; I just create volumes based on what makes sense for the applications hosted on the servers.
- bluedino 5y agoI'd like to see a database benchmark ran instead of a software build.
- jdblair 5y agoOh, this takes me back. I used to spread my swap out across all my disks on my system. When I had 2 disks. I put /boot, / and /var on one disk and /home on the other. When I had more disks, I moved /var onto its own disk, and had an extra drive that I symlinked into /home. I put swap first on all the partitions. It's not like I did any benchmarking, there was just lore that swap should be close to the middle, followed by frequently accessed user data. At some point I got enough RAM that the swap wasn't really important, but I always provisioned it. Now everything is SSD, and I feel like the whole idea of filesystem that you have to mount and keep consistent is kind old fashioned, but we have so much stuff built on the filesystem it will be with us a long time.
- natmaka 5y agoOn a related perspective zswap (Linux) is surprisingly efficient if the system isn't CPU-bound. It is "a Linux kernel feature that provides a compressed write-back cache for swapped pages, as a form of virtual memory compression. Instead of moving memory pages to a swap device when they are to be swapped out, zswap performs their compression and then stores them into a memory pool dynamically allocated in the system RAM" https://en.wikipedia.org/wiki/Zswap https://en.wikipedia.org/wiki/Zswap
- egberts1 5y agoSwap placements that I do: 1. choose the fastest HDD device 2. Use direct partition, no LVM 3. partition in middle of spinning platter’ busiest region of hard drive 4. single swap partition only 5. keep swap and hibernate storage separate. 6. encrypt swap (only downside)
- flatiron 5y agoWonder if zswap would make any difference here?
- zepearl 5y agoQuestion not related to the article: does anybody have hints or a link to some page explaining how to set up Linux so that it uses swap reaaally only if there is almost no free RAM available? I have a few private servers & VMs, all having swap enabled, and all start using swap if I do a lot of I/O even if I have e.g. more than 20GBs free out of 36 being available. Usually swap is not being used just after having booted the server or VM, but after a few hours or days of doing reads & writes to disk the kernel will start writing stuff to swap - it's very little (few KBs being written every few seconds), but that accumulates and after a few days I end up having GBs of swap used. On one hand I just personally hate seeing that happening, on the other hand some of my workloads are irregular so when the workload changes the swap is emptied (at least partially) and the whole thing starts over again. So far I played with the values of "/proc/sys/vm/swappiness" (tried to set there 0, 1, 60, 100) and "/proc/sys/vm/vfs_cache_pressure" (tried to set there 50, 100, 200), but when doing a lot of I/O the OS always ended up using swap. I would like to have swap available/enabled to cover potential extreme cases without having the programs crash (e.g. I might set memory limits of SW that might rarely run concurrently too high, or some database might suddenly allocate more than expected, etc...) => seeing that swap is being/was used would tell me that something is NOK in relation to the total RAM being used by my SW.. .
- callesgg 5y agoThe less the reader arm has to move the faster seeking should be. So if you place the swap near the rest of the files the hdd arm will not need to move so much. Given that this was pretty much a clean Linux install I would assume that most files where at the start of the disk close to the best swap location.
- 10GBps 5y agoWell known but still interesting. Nowadays I generally don't use any swap at all and find it annoying when distros/Windows create swap anyway. I mean if my 128GB+ or even 32GB of primary memory runs out, is it really going to help to swap 2GB to disk? And any larger swap than that is too slow to be usable.