8 ms·

NUMA Deep Dive Part 1: From UMA to NUMA

100 points by wila 10y ago

av500 10y ago
I was expecting more Dirk Pitt in this post
- icc97 10y ago
  I was going to ask that - but assumed I'd get down voted into the err.. Pacific Vortex. I'll down vote myself for that one.
deleted 10y ago
[deleted]
praveenster 10y ago
Nicely written article with good detail about the problems in the older architectures. One thing that I'm still a little fuzzy about is on the Linux kernel side of things in the context of multi core multi socket. Which CPU does the kernel actually run on and how does it dispatch the process/threads on the different cores/sockets?
- wila 10y ago
  Yes, Frank has a very nice series going there. More parts are expected to follow. As to your question on what CPU the kernel normally runs on? AFAIK, it normally starts at core 0, processor 0. You can see for yourself where processes run. For example using the ps command. It will print kernel threads in square brackets and it can also list cpu and core number behind a /. Eg, example output of a linux VM with 2 processors. $ ps -ax PID TTY STAT TIME COMMAND 1 ? Ss 0:03 /sbin/init splash 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 5 ? S< 0:00 [kworker/0:0H] 7 ? S 0:15 [rcu_sched] 8 ? S 0:00 [rcu_bh] 9 ? S 0:00 [migration/0] 10 ? S 0:00 [watchdog/0] 11 ? S 0:00 [watchdog/1] etcetera.. removed the rest for readability The CPU scheduler then handles the rest to see on which thread/core to run a new process. This post might also help: http://superuser.com/questions/389161/what-do-the-mean-in-ps-and-how-do-you-create-a-process-like-it http://superuser.com/questions/389161/what-do-the-mean-in-ps...
- angry_octet 10y ago
  The simple answer is that the kernel runs on all cores. But the answer is really that it depends which part of the kernel you are talking about. Whenever user mode does a trap to supervisor mode it is running the top half of the kernel. An interrupt (device or timer) causes the core to run the bottom half of the kernel. Many systems used to only send these to cpu0, and it is typical that lots of network I/O would occur on cpu0. In the PIII era, with a wide bus shared by all cores, most kernel data structures were not multithreaded, i.e. single run queue, great big kernel locks. There are now many per-core data structures, which can be in local memory. See http://www.tldp.org/LDP/tlk/kernel/kernel.html http://www.tldp.org/LDP/tlk/kernel/kernel.html and for a recent discussion of scheduling: http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf
bogomipz 10y ago
What a fantastic series. The author really put a lot of work into these. Thank you Frank.
- frankdenneman 10y ago
  My pleasure! Im aiming to release some of the articles focussing on ESXi level scheduling this week. Stay tuned!
frankdenneman 10y ago
Thanks all for visiting the articles. Long time HN reader, never expected to appear on HN! WOW!
- angry_octet 10y ago
  Great article. Are those diagrams from Omnigraffle? Obviously you are focusing on x86, but it might be worth mentioning where many of these ideas developed. For example, SGI Origin hardware (http://www.sgidepot.co.uk/origin/isca.pdf http://www.sgidepot.co.uk/origin/isca.pdf).
rconti 10y ago
I'll have to devote some time to reading these. I'm running some 8-socket Oracle boxes and I've got NUMA disabled due to the sheer complexity of maintaining/understanding the configuration. Processor pinning, and so on. I need to educate myself to understand the consequences of this decision, what the payoff would be to enabling NUMA, what my configs should look like, if there's an "easy way out", etc.
- angry_octet 10y ago
  When you say 'NUMA disabled' do you mean NUMA aware scheduling and memory allocation is disabled? Because it is going to be NUMA no matter what you do.
  - rconti 10y ago
    Yes. Supposedly Oracle recommends such a configuration, but I haven't dug into it much beyond that.
    - angry_octet 10y ago
      https://kevinclosson.net/category/oracle-numa/ https://kevinclosson.net/category/oracle-numa/
ythl 10y ago
I wonder how NUMA compares to MAYA HEAP as far as memory performance goes?
- detaro 10y ago
  What is MAYA HEAP?
frankdenneman 10y ago
Updated the series with ESX NUMA constructs: http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esxi-vmkernel-numa-constructs/ http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esx...

100 points by wila 10y ago

av500 10y ago

I was expecting more Dirk Pitt in this post

icc97 10y ago
I was going to ask that - but assumed I'd get down voted into the err.. Pacific Vortex. I'll down vote myself for that one.

deleted 10y ago

[deleted]

praveenster 10y ago

Nicely written article with good detail about the problems in the older architectures. One thing that I'm still a little fuzzy about is on the Linux kernel side of things in the context of multi core multi socket. Which CPU does the kernel actually run on and how does it dispatch the process/threads on the different cores/sockets?

wila 10y ago
Yes, Frank has a very nice series going there. More parts are expected to follow. As to your question on what CPU the kernel normally runs on? AFAIK, it normally starts at core 0, processor 0. You can see for yourself where processes run. For example using the ps command. It will print kernel threads in square brackets and it can also list cpu and core number behind a /. Eg, example output of a linux VM with 2 processors. $ ps -ax PID TTY STAT TIME COMMAND 1 ? Ss 0:03 /sbin/init splash 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 5 ? S< 0:00 [kworker/0:0H] 7 ? S 0:15 [rcu_sched] 8 ? S 0:00 [rcu_bh] 9 ? S 0:00 [migration/0] 10 ? S 0:00 [watchdog/0] 11 ? S 0:00 [watchdog/1] etcetera.. removed the rest for readability The CPU scheduler then handles the rest to see on which thread/core to run a new process. This post might also help: http://superuser.com/questions/389161/what-do-the-mean-in-ps-and-how-do-you-create-a-process-like-it http://superuser.com/questions/389161/what-do-the-mean-in-ps...
angry_octet 10y ago
The simple answer is that the kernel runs on all cores. But the answer is really that it depends which part of the kernel you are talking about. Whenever user mode does a trap to supervisor mode it is running the top half of the kernel. An interrupt (device or timer) causes the core to run the bottom half of the kernel. Many systems used to only send these to cpu0, and it is typical that lots of network I/O would occur on cpu0. In the PIII era, with a wide bus shared by all cores, most kernel data structures were not multithreaded, i.e. single run queue, great big kernel locks. There are now many per-core data structures, which can be in local memory. See http://www.tldp.org/LDP/tlk/kernel/kernel.html http://www.tldp.org/LDP/tlk/kernel/kernel.html and for a recent discussion of scheduling: http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf

bogomipz 10y ago

What a fantastic series. The author really put a lot of work into these. Thank you Frank.

frankdenneman 10y ago
My pleasure! Im aiming to release some of the articles focussing on ESXi level scheduling this week. Stay tuned!

frankdenneman 10y ago

Thanks all for visiting the articles. Long time HN reader, never expected to appear on HN! WOW!

angry_octet 10y ago
Great article. Are those diagrams from Omnigraffle? Obviously you are focusing on x86, but it might be worth mentioning where many of these ideas developed. For example, SGI Origin hardware (http://www.sgidepot.co.uk/origin/isca.pdf http://www.sgidepot.co.uk/origin/isca.pdf).

rconti 10y ago

I'll have to devote some time to reading these. I'm running some 8-socket Oracle boxes and I've got NUMA disabled due to the sheer complexity of maintaining/understanding the configuration. Processor pinning, and so on. I need to educate myself to understand the consequences of this decision, what the payoff would be to enabling NUMA, what my configs should look like, if there's an "easy way out", etc.

angry_octet 10y ago
When you say 'NUMA disabled' do you mean NUMA aware scheduling and memory allocation is disabled? Because it is going to be NUMA no matter what you do.
- rconti 10y ago
  Yes. Supposedly Oracle recommends such a configuration, but I haven't dug into it much beyond that.
  - angry_octet 10y ago
    https://kevinclosson.net/category/oracle-numa/ https://kevinclosson.net/category/oracle-numa/

ythl 10y ago

I wonder how NUMA compares to MAYA HEAP as far as memory performance goes?

detaro 10y ago
What is MAYA HEAP?

frankdenneman 10y ago

Updated the series with ESX NUMA constructs: http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esxi-vmkernel-numa-constructs/ http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esx...