5 ms·

So at one point we where doing scale testing for our product where we needed to simulate systems running our software connected back to a central point. The ide

So at one point we where doing scale testing for our product where we needed to simulate systems running our software connected back to a central point. The idea was to run as many docker containers as we could on a server with 2x24 core and 512GB of RAM. The RAM needed for each container was very small. No matter what the system would start to break around ~1000 containers (this was 4 years ago). After doing may hours the normal debugging we did not see anything on the network stack or linux limits side that we had not already tweaked (so we thought).So out comes strace! Bingo! We found out that the system could not handle the ARP cache with so many end points. Playing with net.ipv4.neigh.default.gc_interval and the stuff associated with it got us up to 2500+ containers.

ktpsns 7y ago
512GB RAM/2500 containers is still 500MB per container. In former days™ this was enough for a computer to run a complete desktop environment with a web browser and 20 tabs open (source: I had a PC with physically 500MB RAM). Is this really the limit for such a decent equipped machine? (I guess a server grade 48 cores, 512GB RAM should be less then 5kEUR nowadays)
- jjeaff 7y ago
  >512GB / 2500 is still 500mb per container Am I missing something? 512gb / 2500 is around 200mb per container.
  - Mirioron 7y ago
    I think he miscalculated. His point still stands, just not with 20 browser tabs.
    - hobs 7y ago
      The important bit is "running our software" - depending on the complexity of said software, 200MB is pretty meh.
      - codegladiator 7y ago
        wait, i had windows xp with firefox and all at 256 mb ram ! which i got later upgraded to 384 mbs because i reused another 128 mb ram chip :)
        philsnow 7y ago
        In ~2003, I replaced my desktop that had been Intel-based with a Duron 800MHz system, only I didn't have enough budget to get it the RAM it required (new/different slot iirc), so I only had the 128MB it came with (whereas my old machine had 768MB cobbled together from like six dimms). I figured that one hop over 100Mbit Ethernet to remote memory was going to be faster then swapping to spinning rust (remember this was before consumer SSDs, and onions on our belts), so I made a ramdisk on the old machine and mounted it over the network with the nbd (network block device) kernel driver, ran swapon on the nbd and boom, extra "512MB" of RAM. It worked amazingly well, and (knock on wood) none of my roommates ever tripped over the Ethernet cables.
        u6eexrtxjjxjexr 7y ago
        So you're the guy who actually managed to "download more RAM". Congrats! On a more serious note, gonna add that trick to my book - still plenty of rust spinnin' round
        paulmd 7y ago
        Infiniband QDR adapters are amazingly cheap and RDMA-aware software can use DMA to poke directly at memory or devices in the other system.
      - myrandomcomment 7y ago
        Yes. Needed ~150MB per container. It was just simulating end points - ie, TCP connections back to a central system.
- u801e 7y ago
  What was the remaining 12 MB of RAM used for?
- Ao7bei3s 7y ago
  He said "The RAM needed for each container was very small" - the RAM is _not_ the limit here. The point is that running many containers is a very different (hard to compare directly) type of load than the main, well optimized use case of running a single large desktop environment. Networking in particular: with thousands of containers, now there are lots more interfaces, routes, conntrack entries, "background" traffic, iptables rules etc. Some problems are algorithmic: for example historically a lot of code has been written with the assumption that the number of network interfaces is small. With >1k interfaces, suddenly O(n) lookups take time. Similarly, iptables rules are run sequentially. etc. Some resources have limits. Exceeding these may impact performance. In his case the load blew up the ARP cache. Nice!
- viraptor 7y ago
  You're assuming that all the memory was used, even though it's not specified in the post. 512GB/2500 is the upper limit. Anything between 0 and that could be the case.
- deleted 7y ago
  [deleted]
Ao7bei3s 7y ago
Nice! But how do you tell that from strace? Wouldn't it just show the guests stuck in connect()? With perf, sure, but strace?
- myrandomcomment 7y ago
  It’s has been awhile, but IIRC we saw the net process for ARP being called and getting stuck.

ktpsns 7y ago

512GB RAM/2500 containers is still 500MB per container. In former days™ this was enough for a computer to run a complete desktop environment with a web browser and 20 tabs open (source: I had a PC with physically 500MB RAM). Is this really the limit for such a decent equipped machine? (I guess a server grade 48 cores, 512GB RAM should be less then 5kEUR nowadays)

jjeaff 7y ago
>512GB / 2500 is still 500mb per container Am I missing something? 512gb / 2500 is around 200mb per container.
- Mirioron 7y ago
  I think he miscalculated. His point still stands, just not with 20 browser tabs.
  - hobs 7y ago
    The important bit is "running our software" - depending on the complexity of said software, 200MB is pretty meh.
    - codegladiator 7y ago
      wait, i had windows xp with firefox and all at 256 mb ram ! which i got later upgraded to 384 mbs because i reused another 128 mb ram chip :)
      - philsnow 7y ago
        In ~2003, I replaced my desktop that had been Intel-based with a Duron 800MHz system, only I didn't have enough budget to get it the RAM it required (new/different slot iirc), so I only had the 128MB it came with (whereas my old machine had 768MB cobbled together from like six dimms). I figured that one hop over 100Mbit Ethernet to remote memory was going to be faster then swapping to spinning rust (remember this was before consumer SSDs, and onions on our belts), so I made a ramdisk on the old machine and mounted it over the network with the nbd (network block device) kernel driver, ran swapon on the nbd and boom, extra "512MB" of RAM. It worked amazingly well, and (knock on wood) none of my roommates ever tripped over the Ethernet cables.
        u6eexrtxjjxjexr 7y ago
        So you're the guy who actually managed to "download more RAM". Congrats! On a more serious note, gonna add that trick to my book - still plenty of rust spinnin' round
        paulmd 7y ago
        Infiniband QDR adapters are amazingly cheap and RDMA-aware software can use DMA to poke directly at memory or devices in the other system.
    - myrandomcomment 7y ago
      Yes. Needed ~150MB per container. It was just simulating end points - ie, TCP connections back to a central system.
u801e 7y ago
What was the remaining 12 MB of RAM used for?
Ao7bei3s 7y ago
He said "The RAM needed for each container was very small" - the RAM is _not_ the limit here. The point is that running many containers is a very different (hard to compare directly) type of load than the main, well optimized use case of running a single large desktop environment. Networking in particular: with thousands of containers, now there are lots more interfaces, routes, conntrack entries, "background" traffic, iptables rules etc. Some problems are algorithmic: for example historically a lot of code has been written with the assumption that the number of network interfaces is small. With >1k interfaces, suddenly O(n) lookups take time. Similarly, iptables rules are run sequentially. etc. Some resources have limits. Exceeding these may impact performance. In his case the load blew up the ARP cache. Nice!
viraptor 7y ago
You're assuming that all the memory was used, even though it's not specified in the post. 512GB/2500 is the upper limit. Anything between 0 and that could be the case.
deleted 7y ago
[deleted]

Ao7bei3s 7y ago

Nice! But how do you tell that from strace? Wouldn't it just show the guests stuck in connect()? With perf, sure, but strace?

myrandomcomment 7y ago
It’s has been awhile, but IIRC we saw the net process for ARP being called and getting stuck.