4 ms·

Great to see the NUMA balancing in. My question has always been what workloads require NUMA balancing in the first place? If I present the kernel with the same

Great to see the NUMA balancing in. My question has always been what workloads require NUMA balancing in the first place? If I present the kernel with the same number of threads as cores and keep all data local to a thread would the existing approach of allocating on the NUMA node the thread is running on have been enough?

fsaintjacques 13y ago
When your process use more than half of the memory, e.g. DBs.
- arielweisberg 13y ago
  The kernel will balance loaded threads across cores so with a policy of allocating off the local NUMA node you actually end up with balanced allocations in practice if you run shared nothing thread per core. In memory databases are my day job so I am pretty interested in cases where things go south because memory isn't balanced. To date it appears like no special actions were necessary, stuff just ends up balanced across nodes. That's why it would be great of someone could characterize when balancing is necessary outside of obvious cases like allocating an entire buffer pool from one thread.
  - pmenon 13y ago
    You should take a look at "OLTP on Hardware Islands" in VLDB 2012: http://vldb.org/pvldb/vol5/p1447_danicaporobic_vldb2012.pdf http://vldb.org/pvldb/vol5/p1447_danicaporobic_vldb2012.pdf
    - MichaelGG 13y ago
      It applies to more than DBs. Running VoIP software, we found that just by setting CPU affinity, we got a major increase in performance. The software in question, FreeSWITCH, is inanely-threaded in a misguided believe that "more threads=more performance" (well that, and it's also just easier to program). When there's thousands of threads going, keeping them and their data local to one NUMA node or less really makes a huge difference.
  - fsaintjacques 13y ago
    The problem is not the balanced memory allocation of the said process, but the kernel evicting IO caches: https://groups.google.com/forum/#!topic/fa.linux.kernel/4IKYjqgROTg https://groups.google.com/forum/#!topic/fa.linux.kernel/4IKY... We experienced this bug in production on hardware where the cost of accessing the other node is exactly over 20 (the limit defined in the kernel). node distances: node 0 1 0: 10 21 1: 21 10
diegocg 13y ago
This article explains the NUMA improvements in this release https://lwn.net/Articles/568870/ https://lwn.net/Articles/568870/

fsaintjacques 13y ago

When your process use more than half of the memory, e.g. DBs.

arielweisberg 13y ago
The kernel will balance loaded threads across cores so with a policy of allocating off the local NUMA node you actually end up with balanced allocations in practice if you run shared nothing thread per core. In memory databases are my day job so I am pretty interested in cases where things go south because memory isn't balanced. To date it appears like no special actions were necessary, stuff just ends up balanced across nodes. That's why it would be great of someone could characterize when balancing is necessary outside of obvious cases like allocating an entire buffer pool from one thread.
- pmenon 13y ago
  You should take a look at "OLTP on Hardware Islands" in VLDB 2012: http://vldb.org/pvldb/vol5/p1447_danicaporobic_vldb2012.pdf http://vldb.org/pvldb/vol5/p1447_danicaporobic_vldb2012.pdf
  - MichaelGG 13y ago
    It applies to more than DBs. Running VoIP software, we found that just by setting CPU affinity, we got a major increase in performance. The software in question, FreeSWITCH, is inanely-threaded in a misguided believe that "more threads=more performance" (well that, and it's also just easier to program). When there's thousands of threads going, keeping them and their data local to one NUMA node or less really makes a huge difference.
- fsaintjacques 13y ago
  The problem is not the balanced memory allocation of the said process, but the kernel evicting IO caches: https://groups.google.com/forum/#!topic/fa.linux.kernel/4IKYjqgROTg https://groups.google.com/forum/#!topic/fa.linux.kernel/4IKY... We experienced this bug in production on hardware where the cost of accessing the other node is exactly over 20 (the limit defined in the kernel). node distances: node 0 1 0: 10 21 1: 21 10

diegocg 13y ago

This article explains the NUMA improvements in this release https://lwn.net/Articles/568870/ https://lwn.net/Articles/568870/