7 ms·

Hmm, that java process has 0.227 TB of mapped virtual memory and 3090% CPU..

dspillett 11y ago
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries can make such readings noticably imprecise (though for CPU time usually in the downwards direction by missing tasks that started+worked+ended between readings). The 233GB of memory seems high but there are possible explanations for this. A server with 32+ cores is not unlikely to have a lot of RAM too, the motherboard in my home server supports up to 128GB so perhaps that large java process genuinely does use more then 200. Also all that memory might not really be in use: it could have been allocated but never accessed so it isn't yet holding pages in RAM or swap for all of it.
- drewg123 11y ago
  The 233GB is VIRT, meaning virtual memory. It need not all be backed by physical memory. For example, if you mmap a file, and then access only a small portion. The RES column shows how much memory is resident (eg, currently backed by physical memory), and that is a much more reasonable 12GB.
- mentat 11y ago
  Probably r3.8xlarge.
  - darkr 11y ago
    jinx!
- darkr 11y ago
  The r3.8xlarge instance type has 32 vCPUs and 244GiB RAM
- myth_buster 11y ago
  For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Yes, the article states that The %CPU column is the total across all CPUs; 1591% shows that that java processes is consuming almost 16 CPUs.
akerro 11y ago
and they run it as root
- to3m 11y ago
  It could be running in a container?
pixelbeat 11y ago
Note that's virtual memory. For determining real RAM usage for programs, I find https://github.com/pixelb/ps_mem https://github.com/pixelb/ps_mem extremely useful
jandrese 11y ago
I'm guessing that the java process is their video server. It's probably having to DRM and bandwidth regulate hundreds of HD streams. Unfortunately, this might mean that all of the stuff he talked about in here didn't actually help him solve the problem because none of it had any visibility inside of their Java VM.
- anon3423423422 11y ago
  Some of their recommendation algorithms run on large boxes in java.

You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service. For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries can make such readings noticably imprecise (though for CPU time usually in the downwards direction by missing tasks that started+worked+ended between readings). The 233GB of memory seems high but there are possible explanations for this. A server with 32+ cores is not unlikely to have a lot of RAM too, the motherboard in my home server supports up to 128GB so perhaps that large java process genuinely does use more then 200. Also all that memory might not really be in use: it could have been allocated but never accessed so it isn't yet holding pages in RAM or swap for all of it.

drewg123 11y ago
The 233GB is VIRT, meaning virtual memory. It need not all be backed by physical memory. For example, if you mmap a file, and then access only a small portion. The RES column shows how much memory is resident (eg, currently backed by physical memory), and that is a much more reasonable 12GB.
mentat 11y ago
Probably r3.8xlarge.
- darkr 11y ago
  jinx!
darkr 11y ago
The r3.8xlarge instance type has 32 vCPUs and 244GiB RAM
myth_buster 11y ago
For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them. Yes, the article states that The %CPU column is the total across all CPUs; 1591% shows that that java processes is consuming almost 16 CPUs.

akerro 11y ago

and they run it as root

to3m 11y ago
It could be running in a container?

pixelbeat 11y ago

Note that's virtual memory. For determining real RAM usage for programs, I find https://github.com/pixelb/ps_mem https://github.com/pixelb/ps_mem extremely useful

jandrese 11y ago

I'm guessing that the java process is their video server. It's probably having to DRM and bandwidth regulate hundreds of HD streams. Unfortunately, this might mean that all of the stuff he talked about in here didn't actually help him solve the problem because none of it had any visibility inside of their Java VM.

anon3423423422 11y ago
Some of their recommendation algorithms run on large boxes in java.