11 ms·

One of the things that makes Ruby so difficult to profile is that in order to do anything really fun (like get memory usage, for instance), you need to shell ou

One of the things that makes Ruby so difficult to profile is that in order to do anything really fun (like get memory usage, for instance), you need to shell out, which means you're forking/doubling your memory footprint immediately, which means you probably can't do this in production. This is especially frustrating because by far the biggest issue you'll have with a Ruby (especially Rails) app is memory usage, so trying to pinpoint what of the many things that are currently running on your box are blowing up your memory can be non-trivial.

semiquaver 9y ago
On linux you can get a process' memory usage by reading the `/proc` pseudo-filesystem. See [0], which uses that approach when supported. You're correct that production ruby applications should try to avoid calling fork() at runtime, which excludes all the standard library mechanisms for shelling out. posix-spawn [1] is an alternative that works well. [0] https://github.com/schneems/get_process_mem https://github.com/schneems/get_process_mem [1] https://github.com/rtomayko/posix-spawn https://github.com/rtomayko/posix-spawn
jvns 9y ago
Why do you have to fork to get a memory profile? I don't know anything about memory profiling yet (I collected a bunch of links in https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b5 https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b... but haven't read their source yet) but I'd like to know more. It seems like it should be possible to get a memory profile in production without a prohibitive amount of overhead though. Also, I believe forking shouldn't actually double memory usage because of copy on write.
- wdewind 9y ago
  Ruby can't tell how much memory it is using from within the process, the only way to is shell out and scrape the output of ps. I would love to be wrong about this, I did a fair amount of research and all the profiling libraries I read used this method. One thing I've done instead is use object allocation, which is available from the builtin benchmarker, as a proxy for memory usage. I don't have a good sense of how accurate that is, but anecdotally it seemed useful and correlated to stuff like page load times. (Edit: ^happily I am wrong about this, though I believe the copy on write thing below is still correct, see further discussion) Copy on write does definitely help here, only for Ruby > 2.0.
  - anarazel 9y ago
    > the only way to is shell out and scrape the output of ps Why couldn't you just read /proc/self/status and /proc/self/smaps in process? Would be like an order of magnitude less expensive.
    - wdewind 9y ago
      Because I have a shitty knowledge of Unix :) Best way to get the right answer on the internet is to post the wrong one.
      - marios 9y ago
        For what it's worth, the /proc filesystem is not on all UNIX /UNIX-like OSes. I believe it originated in Linux, and FreeBSD has a procfs interface as well, although it does not have all the bells and whistles you will find in Linux. OpenBSD on the other hand, does not have procfs at all.
  - TheDong 9y ago
    I don't have much experience in ruby profiling, but what you're saying doesn't make sense. > the only way to is shell out and scrape the output of ps All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. That isn't a useful amount of memory because, well, it doesn't actually show where object allocations go. > Copy on write does definitely help here, only for Ruby > 2.0. I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. I really don't understand what you're trying to say or why you mention ps at all.
    - wdewind 9y ago
      > All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. Yep, great point, I am simply wrong about this. > I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. Re: Copy on write the way Ruby implements GC in < 2.0 screws this up: > The way ruby creates objects, the GC flag or reserved bit is stored in the object itself. So, as you would have guessed by now, when the GC runs in one of the processes, the GC flag would be modified in all the objects even if they are present in the shared pool. Now, the OS would sense this and trigger a copy-on-write making private copies of the objects in each child’s memory space. https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-copy-on-write-f5eddef21485 https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...

semiquaver 9y ago

On linux you can get a process' memory usage by reading the `/proc` pseudo-filesystem. See [0], which uses that approach when supported. You're correct that production ruby applications should try to avoid calling fork() at runtime, which excludes all the standard library mechanisms for shelling out. posix-spawn [1] is an alternative that works well. [0] https://github.com/schneems/get_process_mem https://github.com/schneems/get_process_mem [1] https://github.com/rtomayko/posix-spawn https://github.com/rtomayko/posix-spawn

jvns 9y ago

Why do you have to fork to get a memory profile? I don't know anything about memory profiling yet (I collected a bunch of links in https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b5 https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b... but haven't read their source yet) but I'd like to know more. It seems like it should be possible to get a memory profile in production without a prohibitive amount of overhead though. Also, I believe forking shouldn't actually double memory usage because of copy on write.

wdewind 9y ago
Ruby can't tell how much memory it is using from within the process, the only way to is shell out and scrape the output of ps. I would love to be wrong about this, I did a fair amount of research and all the profiling libraries I read used this method. One thing I've done instead is use object allocation, which is available from the builtin benchmarker, as a proxy for memory usage. I don't have a good sense of how accurate that is, but anecdotally it seemed useful and correlated to stuff like page load times. (Edit: ^happily I am wrong about this, though I believe the copy on write thing below is still correct, see further discussion) Copy on write does definitely help here, only for Ruby > 2.0.
- anarazel 9y ago
  > the only way to is shell out and scrape the output of ps Why couldn't you just read /proc/self/status and /proc/self/smaps in process? Would be like an order of magnitude less expensive.
  - wdewind 9y ago
    Because I have a shitty knowledge of Unix :) Best way to get the right answer on the internet is to post the wrong one.
    - marios 9y ago
      For what it's worth, the /proc filesystem is not on all UNIX /UNIX-like OSes. I believe it originated in Linux, and FreeBSD has a procfs interface as well, although it does not have all the bells and whistles you will find in Linux. OpenBSD on the other hand, does not have procfs at all.
- TheDong 9y ago
  I don't have much experience in ruby profiling, but what you're saying doesn't make sense. > the only way to is shell out and scrape the output of ps All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. That isn't a useful amount of memory because, well, it doesn't actually show where object allocations go. > Copy on write does definitely help here, only for Ruby > 2.0. I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. I really don't understand what you're trying to say or why you mention ps at all.
  - wdewind 9y ago
    > All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using. Yep, great point, I am simply wrong about this. > I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed. Re: Copy on write the way Ruby implements GC in < 2.0 screws this up: > The way ruby creates objects, the GC flag or reserved bit is stored in the object itself. So, as you would have guessed by now, when the GC runs in one of the processes, the GC flag would be modified in all the objects even if they are present in the shared pool. Now, the OS would sense this and trigger a copy-on-write making private copies of the objects in each child’s memory space. https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-copy-on-write-f5eddef21485 https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...