6 ms·
Swap is still actively harmful if placed on an unsuitable device. A few days ago, I got a complaint about slow ops in a Ceph cluster, and one of the Ceph OSDs
by patrakov 2y ago
Swap is still actively harmful if placed on an unsuitable device.
A few days ago, I got a complaint about slow ops in a Ceph cluster, and one of the Ceph OSDs spent quite a visible amount of time in response to a simple "ceph tell osd.13 ops" query that is a natural first step when debugging slow ops. Upon investigation, it was found that one of the nodes had a significantly higher load average (50) vs others (14-18). In "top", out of 256 GB of RAM that the node had, approximately 100 GB were cached and thus formally available, yet, out of 8 GB of swap, 6 GB were used, and kswapd consumed a non-negligible percentage of the CPU time.
The kernel is some 5.15.x Ubuntu kernel; the root disk (which also holds the swap file) is a cheap SATADOM, and SATADOMs, in general (yes I know there are exceptions), have miserable performance and low write endurance; I forgot to check SMART statistics but cannot exclude that the drive might have started failing. I did check the dmesg, though, and there were no I/O errors.
The problem has been resolved by disabling the swap file. I should probably have set up swap on ZRAM instead, but a configuration without swap also works.
- martinsnow 2y agoThat's a configuration issue on your part. You shouldn't put SWAP on devices that has basically no throughput.
- patrakov 2y agoThat's on the customer's part, or, more properly said, on the Ubuntu installer's part. Defaults matter.
- kristopolous 2y ago> Swap is still actively harmful if placed on an unsuitable device. Reading that, I just realized it could be done over the network - that would be impressively glacial. Should be a good laugh. Heck, why not over a db9 null modem slip connection? Start a compile and send it off to the Long Now Foundation - set it up next to John Cage's ASLSP organ.
- SSLy 2y agoswap would be fast enough on NVMe-oF for anything not a DB, no?
- kristopolous 2y agoI'd imagine latency matters more than throughout. Complicated disk operations over my 10Gb fiber network still sucks like it did over 10baseT 30 years ago. Maybe there's smarter ways to set things up but NFS/sshfs is still bogged down for me. 1/5th - ~1/100th depending on the application. Also I don't have fancy money. I was imagining the $20 gear on Amazon/eBay. Maybe that's my problem...
- SSLy 2y agoOh, I've meant putting NVMe proto on a wire, with such drives on the other side. Not traditional network app protocols.
- rwmj 2y agoThat's the vision of CXL :-) To be fair, the network is very very fast. https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link
- kristopolous 2y agoOh this is a special cluster interconnect. Yes, those have always been impressive and usually cost as much as a luxury sports car. Hopefully the increased demand from the AI world will help bring those prices down.
- KaiserPro 2y agoinfiniband is pretty cheap now!
- DEADMINCE 2y agoApps would crash before the device got the traffic back telling it what to swap out.
- worthless-trash 2y agoI just searched to see what Ceph reccomended, and here is the funny part: -- Configuring the operating system with swap to provide additional virtual memory for daemons is not advised for modern systems. Doing may result in lower performance, and your Ceph cluster may well be happier with a daemon that crashes vs one that slows to a crawl. -- If a single ceph node crashes and restarts, I assume that the cluster will pick up the slack and heal/deal?
- patrakov 2y agoCorrect. A node that crashed quickly (or where earlyoom killed something - I am looking at you, https://tracker.ceph.com/issues/48673 https://tracker.ceph.com/issues/48673) is visible only as a monitoring alert that goes away by itself, because all Ceph daemons are configured to be restarted after a crash by default. A slow/swapping node, on the other hand, leads to customer or end user complaints.
- KaiserPro 2y agoBut why do you have only 8 gigs of swap? the OSD's main job is file serving, so you need to use as much ram as possible to act as virtual tiered storage (caveats apply) so having a large swap to hold useful but not often used bits of ram is really useful for performance. Our file servers used to have ram+~30% as swap.
- stefantalpalaru 2y ago[dead]
- antongribok 2y agoCeph actually works very well with swap on zram. I have some older Ceph clusters that have 48GB zram devices used for swap. It sometimes helps with defragmenting memory, memory pressure in general, and depending on your workload and other things running on the box, you could have better performance by being able to have more things in page cache.