4 ms·

Why do you have to fork to get a memory profile? I don't know anything about memory profiling yet (I collected a bunch of links in https://gist.github.com/jvns/

by jvns 9y ago

Why do you have to fork to get a memory profile? I don't know anything about memory profiling yet (I collected a bunch of links in https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b5 https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b... but haven't read their source yet) but I'd like to know more. It seems like it should be possible to get a memory profile in production without a prohibitive amount of overhead though. Also, I believe forking shouldn't actually double memory usage because of copy on write.

wdewind 9y ago
Ruby can't tell how much memory it is using from within the process, the only way to is shell out and scrape the output of ps. I would love to be wrong about this, I did a fair amount of research and all the profiling libraries I read used this method. One thing I've done instead is use object allocation, which is available from the builtin benchmarker, as a proxy for memory usage. I don't have a good sense of how accurate that is, but anecdotally it seemed useful and correlated to stuff like page load times. (Edit: ^happily I am wrong about this, though I believe the copy on write thing below is still correct, see further discussion) Copy on write does definitely help here, only for Ruby > 2.0.
- anarazel 9y ago
  > the only way to is shell out and scrape the output of ps Why couldn't you just read /proc/self/status and /proc/self/smaps in process? Would be like an order of magnitude less expensive.
  - wdewind 9y ago
    Because I have a shitty knowledge of Unix :) Best way to get the right answer on the internet is to post the wrong one.
    - marios 9y ago
      For what it's worth, the /proc filesystem is not on all UNIX /UNIX-like OSes. I believe it originated in Linux, and FreeBSD has a procfs interface as well, although it does not have all the bells and whistles you will find in Linux. OpenBSD on the other hand, does not have procfs at all.
- TheDong 9y ago
  I don't have much experience in ruby profiling, but what you're saying doesn't make sense. > the only way to is shell out and scrape the output of ps All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. That isn't a useful amount of memory because, well, it doesn't actually show where object allocations go. > Copy on write does definitely help here, only for Ruby > 2.0. I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. I really don't understand what you're trying to say or why you mention ps at all.
  - wdewind 9y ago
    > All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. Yep, great point, I am simply wrong about this. > I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. Re: Copy on write the way Ruby implements GC in < 2.0 screws this up: > The way ruby creates objects, the GC flag or reserved bit is stored in the object itself. So, as you would have guessed by now, when the GC runs in one of the processes, the GC flag would be modified in all the objects even if they are present in the shared pool. Now, the OS would sense this and trigger a copy-on-write making private copies of the objects in each child’s memory space. https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-copy-on-write-f5eddef21485 https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...
    - zb 9y ago
      That is a genuine issue with fork(), but if you do fork() followed immediately by exec() then you won't ever encounter it. (The new process's memory space completely replaces the copied pages of the child.) So it's a real problem for e.g. spinning off worker processes, but not at all for e.g. calling out to a shell.
- deleted 9y ago
  [deleted]
- sciurus 9y ago
  Run the code snippet below to prove to yourself that a Ruby process can get its memory usage. print File.readlines('/proc/self/status').select {|line| line =~ /VmRSS/}[0] This is not what people mean when they talk about memory profiling, but it looked like what you were talking about, so I thought I'd share it.
- bm1362 9y ago
  As an aside, I liked your response to being wrong and would totally work with you.
  - wdewind 9y ago
    Hey thanks! In that case, let me shamelessly say I'm actually hiring for iOS and Rails full stack (We are a seed stage startup working on a pretty fun consumer problem, Rails experience not necessary, admitting you are wrong experience helpful) :) Contact info in my profile

by jvns 9y ago

wdewind 9y ago

Ruby can't tell how much memory it is using from within the process, the only way to is shell out and scrape the output of ps. I would love to be wrong about this, I did a fair amount of research and all the profiling libraries I read used this method. One thing I've done instead is use object allocation, which is available from the builtin benchmarker, as a proxy for memory usage. I don't have a good sense of how accurate that is, but anecdotally it seemed useful and correlated to stuff like page load times. (Edit: ^happily I am wrong about this, though I believe the copy on write thing below is still correct, see further discussion) Copy on write does definitely help here, only for Ruby > 2.0.

anarazel 9y ago
> the only way to is shell out and scrape the output of ps Why couldn't you just read /proc/self/status and /proc/self/smaps in process? Would be like an order of magnitude less expensive.
- wdewind 9y ago
  Because I have a shitty knowledge of Unix :) Best way to get the right answer on the internet is to post the wrong one.
  - marios 9y ago
    For what it's worth, the /proc filesystem is not on all UNIX /UNIX-like OSes. I believe it originated in Linux, and FreeBSD has a procfs interface as well, although it does not have all the bells and whistles you will find in Linux. OpenBSD on the other hand, does not have procfs at all.
TheDong 9y ago
I don't have much experience in ruby profiling, but what you're saying doesn't make sense. > the only way to is shell out and scrape the output of ps All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. That isn't a useful amount of memory because, well, it doesn't actually show where object allocations go. > Copy on write does definitely help here, only for Ruby > 2.0. I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. I really don't understand what you're trying to say or why you mention ps at all.
- wdewind 9y ago
  > All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. Yep, great point, I am simply wrong about this. > I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. Re: Copy on write the way Ruby implements GC in < 2.0 screws this up: > The way ruby creates objects, the GC flag or reserved bit is stored in the object itself. So, as you would have guessed by now, when the GC runs in one of the processes, the GC flag would be modified in all the objects even if they are present in the shared pool. Now, the OS would sense this and trigger a copy-on-write making private copies of the objects in each child’s memory space. https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-copy-on-write-f5eddef21485 https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...
  - zb 9y ago
    That is a genuine issue with fork(), but if you do fork() followed immediately by exec() then you won't ever encounter it. (The new process's memory space completely replaces the copied pages of the child.) So it's a real problem for e.g. spinning off worker processes, but not at all for e.g. calling out to a shell.
deleted 9y ago
[deleted]
sciurus 9y ago
Run the code snippet below to prove to yourself that a Ruby process can get its memory usage. print File.readlines('/proc/self/status').select {|line| line =~ /VmRSS/}[0] This is not what people mean when they talk about memory profiling, but it looked like what you were talking about, so I thought I'd share it.
bm1362 9y ago
As an aside, I liked your response to being wrong and would totally work with you.
- wdewind 9y ago
  Hey thanks! In that case, let me shamelessly say I'm actually hiring for iOS and Rails full stack (We are a seed stage startup working on a pretty fun consumer problem, Rails experience not necessary, admitting you are wrong experience helpful) :) Contact info in my profile