3 ms·
Really? I'm not willing to let you wave your arms about that last conclusion. Suppose you had 100 frontends and 10000 backends in a fully-connected manner with
by jeffbee 6y ago
Really? I'm not willing to let you wave your arms about that last conclusion. Suppose you had 100 frontends and 10000 backends in a fully-connected manner with 1 million sockets. You spent 8KiB on this state (I think that's a bit low, but I'm using your figure). That would only be a significant fraction of (more than 1% of) 100 * 1E6 * 8KiB == 800GiB, spread across your 10100 containers, meaning they each have only 80MiB of memory? This seems very unlikely.
A much more likely scenario is each of your 10100 things has at least 2-4GB of memory per CPU core, meaning at least 20-40TiB of memory in your system, meaning that the million connections cost you 0.02% of your system memory, or essentially nothing.
- AtNightWeCode 6y agoMemory when using a workload scheduler can for sure be as low as 80MB. I believe it was set at 100MB for all containers handled by Nomad in a solution I used to work with. It would be insanely expensive to build solutions with your specs to just solve a problem that should be solved by redesigning for fewer connections.
- jeffbee 6y ago36MB is a fatally small container on Linux x86-64, and really nobody should try to use a memcg smaller than 128MB.
- AtNightWeCode 6y agoYeah, not uncommon that containers crashed and the developers came back and increased the amount of memory once Nomad killed stuff.
- mumblemumble 6y agoI think the point of contention here is probably what a typical system looks like. I haven't personally had the pleasure of working on a system that's anywhere close to having 100 frontends and 10,000 backends, and I don't have many friends who have, either, so I assume that's an atypical use case. Perhaps I'm wrong there.
- jeffbee 6y agoThen why would you have a million established connections? That's my whole point here: don't worry about the connection count until you have a concrete need to worry about it.
- mumblemumble 6y agoI think that's actually my whole point here. If I'm running a system on anywhere near the scale I'm used to operating, you damn well better believe that I'm going to worry about "20 billion connections." The concrete reason, which I can give a priori without even waiting for it to happen first, is that the best case scenario for what's going on is probably that the servers are being hugged to death. That and we've violated the laws of physics. Because even by the numbers that you agreed are grossly optimistic (which was deliberate), we're looking at that consuming an order of magnitude or two more RAM than we have across all our servers. So that would admittedly be cool. But still.
- morelisp 6y ago> A much more likely scenario is each of your 10100 things has at least 2-4GB of memory per CPU core This is nearly two orders of magnitude off (250-500MB using 4-16 CPU cores) for some of our most connection-intense services. And as you said 8KiB is extremely low, in practice more like 2-10x that once all buffering is taken into account. >5% of my memory spent on connection handling that (by the original assumption) I don't need, yes, that worries me.
- jeffbee 6y agoThat just doesn't make any sense. You can't even build a computer with 16 cores and 500MB of memory, so why would you have that ratio in production? This whole thread has been completely surreal.
- morelisp 6y ago> You can't even build a computer with 16 cores and 500MB of memory, so why would you have that ratio in production? Because the service has no substantial local state so doesn't need any more than that? Should we invent unnecessary local state just so we can use more memory? That sounds considerably more surreal to me. Or maybe instead we should use all the memory we saved to keep our 200-300GB working set for a different service in RAM colocated, which is what we do...