4 ms·

> "Occurs in prod only, infrequent, no error message, just does nothing." With proper logging, a few days? - pull up bug report - figure out user from logs u

> "Occurs in prod only, infrequent, no error message, just does nothing." With proper logging, a few days? - pull up bug report - figure out user from logs usually based on what he was doing plus IP address + time issue ocurred - look at what he was doing and what went wrong - look at code, see what could possibly go wrong, fix all found issues, apply patch, test and deploy. (alternative - see what recent changes were done there from last few deploys to find possible culprits) Now imagine small issues at scale, millions of users, lots of servers and services, complex architectures... where you can't even reproduce. Those are fixable with logs. Debugger won't really work unless you get a crash dump. Memory snapshot as you say might work - but to get the snapshot I assume it pauses the service to generate memory dump and takes it offline... so equivalent to crash/assert?, otherwise I'd be very curious how to get instant snapshots :)

jiggawatts 4y ago
Essentially it forks the process. The original “keeps going” unaffected, the fork just sits there doing nothing as its memory is written out to disk and then killed. It’s not perfect, but it’s close enough to be indistinguishable in practice. It “feels” like you attached the debugger to the production process. You get the heap, the call stack of every thread, etc… It makes it trivial to inspect things like “what was the database query and what result set did it return” or “which specific heap object was null”.
- adriancr 4y ago
  > Essentially it forks the process. That is a neat idea, I'll definitely use it in the future where possible. > The original “keeps going” unaffected That's not really the case, copy-on-write pages means any write after the fork requires a new page in ram. That will mean increased memory usage and slowdown of writes in addition to fork costs. On something that uses a lot of memory and writes a lot it will be significant. If the service is already optimized to run with just barely enough ram to keep costs down then this fork-and-dump is going to oom in prod. > It “feels” like you attached the debugger to the production process. Yeah, memory dumps are very helful, gdb + dump usually means bug is fixed barring any corruption that happened long before point of manifestation or business logic issues.

jiggawatts 4y ago

Essentially it forks the process. The original “keeps going” unaffected, the fork just sits there doing nothing as its memory is written out to disk and then killed. It’s not perfect, but it’s close enough to be indistinguishable in practice. It “feels” like you attached the debugger to the production process. You get the heap, the call stack of every thread, etc… It makes it trivial to inspect things like “what was the database query and what result set did it return” or “which specific heap object was null”.

adriancr 4y ago
> Essentially it forks the process. That is a neat idea, I'll definitely use it in the future where possible. > The original “keeps going” unaffected That's not really the case, copy-on-write pages means any write after the fork requires a new page in ram. That will mean increased memory usage and slowdown of writes in addition to fork costs. On something that uses a lot of memory and writes a lot it will be significant. If the service is already optimized to run with just barely enough ram to keep costs down then this fork-and-dump is going to oom in prod. > It “feels” like you attached the debugger to the production process. Yeah, memory dumps are very helful, gdb + dump usually means bug is fixed barring any corruption that happened long before point of manifestation or business logic issues.