5 ms·
If you decompress it to an mmaped file it'll be one of the first things written to disk under memory pressure anyways and instantly available in normal situatio
by zamadatix 7y ago
If you decompress it to an mmaped file it'll be one of the first things written to disk under memory pressure anyways and instantly available in normal situations.
With the ever decreasing cost of flash and it's ever increasing speed relative to the CPU compression is not really worth what it used to be to startup times 10 years ago though.
- ceph_ 7y agoSwapping is fine for workstations and home computers. But high performance machines running in a production environments will absolutely have swap disabled. The performance difference between RAM and disk is not an acceptable tradeoff. RAM will be tightly provisioned and jobs will be killed rather than letting the machine OOM.
- Galanwe 7y agoAre you arguing you prefer your production jobs to be killed rather than slowed ?
- sagichmal 7y agoGenerally yes. Swapping changes the perf characteristics of that process (and often any other process on the same machine) in unpredictable ways. It's better to have predictable process termination -- with instrumentation describing what went wrong, so capacity planning and resource quotas can be updated. The process failure would generally be compensated-for at a higher level, anyway.
- Galanwe 7y ago> “process failure would generally be compensated-for at a higher level” I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine? I personally do have swap on all my production machines. I’m not relying on it, but it acts as a safety net in case unexpected memory usage sneak in some jobs. This goes with a sane periodic review of host metrics of course to ensure safety nets do not become the norm. I much prefer a slightly delayed job than a failed job.
- avianlyric 7y agoI guess it depends on the complexity of your distributed system (assuming you’re operating one). We prefer to have job OOM kill and get retried elsewhere (which could be a completely different machine) and we have plenty of infrastructure that makes this trivial. This infrastructure also deals with other types of partial failure, such as complete machine failure. As mentioned above, paging introduces strange pref behaviour. Which may not always be important, but if your working under tight latency requirements then paging can push you over that boundary. That sounds strange when we’re happy to see jobs die entirely (that screws latency). But the issue with paging is you have no idea when it’s gonna hit you, and may impact a job that’s behaving perfectly fine, except something got paged out by a badly behaving job. Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour, and making cause-and-effect highly correlated (oh look thing X just OOMed, probably means thing X consumed too much memory. Rather than thing Y has strange tail latency because thing X keeps consuming too much memory). It’s failing fast, but for memory rather than exceptions.
- Galanwe 7y agoI get your point, and I think we agree here. I mainly wanted to argue against the original statement, which was quite broad “high performance machines running in a production environments will absolutely have swap disabled”. Would you agree to rephrase both our arguments by: - paging incurs seemingly random performance degradation of processes and should be avoided - if you have a form of task queue/job distribution system which handles automatic re-run, and can afford at no business cost to restart a process from scratch, then disabling swapping allows fail fast behaviour - otherwise swapping can be used as a safety net for programs that would be better off slightly late than restarted from scratch - both scenarios require sane monitoring of process behaviours, to catch symptomatic failures/restart in case 1) and recurring swap usage in case 2)
- AdamJacobMuller 7y ago> if your working under tight latency requirements then paging can push you over that boundary Even if you're not under tight requirements, swap can do strange things. I've actually seen situations where hitting swap, even trivially, can cause massive increases in latency. I'm talking about jobs which took 10s of milliseconds to complete now taking multiple 10s of seconds. I've even seem some absurdly bad memory management where Linux will make very very poor choices about what to page out. > Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour 1000% agreed. Fail fast rather than fail slowly.
- Kt4aC1pQVIzY 7y agoIt depends on the memory profile of that particular job of course, and the overall latency/throughput requirements of the application. Swap memory being as slow as it is I've experienced few jobs that can reasonably handle it.
- joemag 7y agoFor those of us who build distributed systems, it’s all about number of modes we need to design for, test, and monitor. If all I have to worry about is process death, I can design my service around that. Monitoring for process death is generally pretty straightforward as well. Gray failures (like process slowdowns), on the other hand, are fairly difficult to design for, and detect. And it can wreak havoc on distributed systems if one of the nodes suffers a gray failure.
- mrmuagi 7y agoWhen slowed is orders of magnitude slower -- then they are effectively being slowly choked to death anyway.
- innagadadavida 7y agoVirtual memory works even without swap enabled. Since the mapped file is the binary, and code is never changed after loading, the OS can simply take pages backing the memory. But when there is a page fault, it will be bought back in.
- bryan_w 7y agoYeah, but why go through all that trouble when you can just add more memory? It's not 1996, machines can have more than 1GB of ram
- naniwaduni 7y agoYou haven't added memory yet because you didn't know your memory requirements before running your prod workload.
- raverbashing 7y agoI don't buy the "never swap" school of thought. Sure, swap is bad on steady state loads. But for transient loads it works Unless you like playing "why did my service got killed this time" every once in a while
- gdxhyrd 7y ago> Unless you like playing "why did my service got killed this time" every once in a while There is no playing: the error is logged and quite clear, by the application or by the kernel. Being consistent, predictable and as deterministic as possible is a key feature of reliable software, and paging into swap kills all that.
- kweinber 7y agoReliable software used swapping for years. Artificially having a low virtual memory ceiling by disabling swap is just austerity for almost no benefit. Paging into swap often has almost no perceptible effect in regular usage unless you are running software with microsecond guarantees on infrequently accessed processes (those that get swapped out).
- koffiezet 7y agoIn 'modern' microservice architectures, this is not true. If you look at the k8s approach, reliable software is created through redundancy. A part of the app being killed by the OOM shouldn't matter, it should automatically be rescheduled on another node. Kubernetes as a platform recommends disabling swap completely, and you have to explicitly allow nodes to have a swap, otherwise it fails. This is sane behaviour if you're dealing with a large cluster with a complex architecture you no single person could or should know all the ins and outs of. There is no "let's log on to the machine and see what's happening" when dealing with this types of architectures, even at smaller scale. And a massive part of Go's target is exactly these modern architectures/workloads...
- raverbashing 7y ago
- zamadatix 7y agoMemory mapped files don't go to swap under pressure they go to the file they were mapped to, no different than the OS knowing you're debug data is in the file just not loaded to memory yet which is the only other scenario here.
- zik 7y agoI think you're confusing memory mapped files with swap. They may be sibling technologies but they're not the same thing at all.