4 ms·

I’ve run clusters of several thousands machines installed with petabytes of ram with no swap (or even disks). It works just fine however you need to keep a app

I’ve run clusters of several thousands machines installed with petabytes of ram with no swap (or even disks). It works just fine however you need to keep a appropriate headroom to allow the kernel to do its thing with caches as indicated otherwise things get very weird very quickly. Containers are very helpful in this regard for helping explicitly divide a machine up between processes without allowing any one to get out of hand.

georgebarnett 7y ago
As an aside, the commentary at the bottom of the post is very insightful and a great example of how important it is to carefully choose what is being optimised for.
z3t4 7y ago
I guess this works if you run few well known apps which manages memory well (yeh right). Maybe recycle/restart in regular intervals. Some programs have 4-8GB limits and will crash by themselves. Or you have a watcher service and recycle/restart after some threshold. And probably your distributed system can handle a few nodes going down. But without memory they will start to act weird. With swap they will usually work, but slower, and might eventually grind to an halt anyway, but it will make it easier to make a graceful shutdown/restart.
- VectorLock 7y ago
  If your application starts running slower it should be killed. Redundancy means not having to worry about graceful anything.
  - jmiserez 7y ago
    You've stated this multiple times in this thread now. Maybe consider that not everyone has the same use case as you: some people are running larger chunks of computation per node or even stateful applications. Even when they could be killed and are redundant, running a bit slower for a moment until they recover may be preferable to restarting the node (which also takes time).
    - VectorLock 7y ago
      There are lots of different application models the issue here is ton of rule-of-thumb regression from "redundancy is important" to "swap is okay after-all I guess."
rambojazz 7y ago
> I’ve run clusters of several thousands machines installed with petabytes of ram Do you mean a cluster of many computers that are collectively seen by the OS as a single machine with petabytes of ram? I can I do that too?
- georgebarnett 7y ago
  Sorry if it’s not clear. The cluster had thousands of individual machines, each performing their own discrete tasks with aggregate ram across the cluster in the petabytes. There was no swap on any of those machines.

georgebarnett 7y ago

As an aside, the commentary at the bottom of the post is very insightful and a great example of how important it is to carefully choose what is being optimised for.

z3t4 7y ago

I guess this works if you run few well known apps which manages memory well (yeh right). Maybe recycle/restart in regular intervals. Some programs have 4-8GB limits and will crash by themselves. Or you have a watcher service and recycle/restart after some threshold. And probably your distributed system can handle a few nodes going down. But without memory they will start to act weird. With swap they will usually work, but slower, and might eventually grind to an halt anyway, but it will make it easier to make a graceful shutdown/restart.

VectorLock 7y ago
If your application starts running slower it should be killed. Redundancy means not having to worry about graceful anything.
- jmiserez 7y ago
  You've stated this multiple times in this thread now. Maybe consider that not everyone has the same use case as you: some people are running larger chunks of computation per node or even stateful applications. Even when they could be killed and are redundant, running a bit slower for a moment until they recover may be preferable to restarting the node (which also takes time).
  - VectorLock 7y ago
    There are lots of different application models the issue here is ton of rule-of-thumb regression from "redundancy is important" to "swap is okay after-all I guess."

rambojazz 7y ago

> I’ve run clusters of several thousands machines installed with petabytes of ram Do you mean a cluster of many computers that are collectively seen by the OS as a single machine with petabytes of ram? I can I do that too?

georgebarnett 7y ago
Sorry if it’s not clear. The cluster had thousands of individual machines, each performing their own discrete tasks with aggregate ram across the cluster in the petabytes. There was no swap on any of those machines.