4 ms·

How would a program detect the use of a debugger? I know a lot of crackmes and other anti-piracy measures involve detecting the use of one but am not sure how t

by jtchang 10y ago

How would a program detect the use of a debugger? I know a lot of crackmes and other anti-piracy measures involve detecting the use of one but am not sure how they do it. Do they just look for running processes with a debugger signature like softice?

colejohnson66 10y ago
The most basic one asks the kernel if there's a debugger attached. Others check if certain programs are running. There's more complex ones, but I can't remember them right now
gst 10y ago
Lots of different approaches: To figure out if a process is ptraced the process can try to ptrace itself. An executable can only get ptraced once at a time, so the second call to ptrace will fail. Of course you can workaround that check by patching the code so that it doesn't call ptrace. Breakpoints are also easy to spot. Just look for 0xCC in the memory. But of course there are also multiple ways to work around those checks (such as using harder to detect hardware breakpoints instead of software breakpoints).
rollulus 10y ago
What I recall from the old days: it was indeed possible to detect Softice, through various ways, for instance simply trying to use its own API (through int3 as well having registers set to special magic values). Another trick to detect breakpoints as an anti-debugging measure was to compute a key over the block of memory that was to be protected. Any int3s in there would give a different output. A countermeasure was to use "memory breakpoints" which work via the x86 debug registers: https://en.wikipedia.org/wiki/X86_debug_register https://en.wikipedia.org/wiki/X86_debug_register
mappu 10y ago
An ideal debugger can't be detected. In the specific case of x86 and int3 bps, you can abuse variable-length instruction encoding to jump into the middle of another instruction. Then code execution can differ depending on the presence of 0xcc.
- userbinator 10y ago
  One common trick was to make the code itself part of a calculation for decrypting/checksumming/encryption/etc., so any modifications would cause a silent failure much later.
  - crististm 10y ago
    The game King's Keep on Sinclair Spectrum was (in a way) like that. It had one or two obfuscated subroutines for xor encoding the main program. They were pretty well localized so once I identified them, decompiling the rest was trivial. Fun times then and now :)
pcwalton 10y ago
> How would a program detect the use of a debugger? IsDebuggerPresent() is the most straightforward way: https://msdn.microsoft.com/en-us/library/windows/desktop/ms680345(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
- umanwizard 10y ago
  The most straightforward way on one particular platform. There are definitely apps using anti-debugger techniques even on iOS.
dracht 10y ago
There's a bunch of ways. Some are obvious (like a call to IsDebuggerPresent()) and some not so much (like tripping some exception that only gets thrown when a process is not being debugged). Youtuber MalwareAnalysisForHedgehogs[0] has some interesting videos about this and anti-reversing tricks in general. https://www.youtube.com/channel/UCVFXrUwuWxNlm6UNZtBLJ-A https://www.youtube.com/channel/UCVFXrUwuWxNlm6UNZtBLJ-A
0x4a42 10y ago
There are many different techniques that you can find in this paper (wich, I think, is a reference on the subject): http://pferrie.host22.com/papers/antidebug.pdf http://pferrie.host22.com/papers/antidebug.pdf For a fresh compilation (which includes the previous paper) you can check this article: http://antukh.com/blog/2015/01/19/malware-techniques-cheat-sheet/ http://antukh.com/blog/2015/01/19/malware-techniques-cheat-s...
j4_james 10y ago
One technique I remember being used in DOS apps from many years ago was that the code would be encrypted in such a way that the next instruction to be executed would only be decrypted immediately before it was run. This was achieved by setting up the single step interrupt as the decrypter, and running the code in single step mode. The fact that the code was encrypted meant the debugger couldn't disassemble it in any meaningful way, and also made it impossible to set a breakpoint (since the breakpoint would just end up being "decrypted" into some other opcode that would inevitably crash). The debugger also couldn't step through the code, because taking over the single step interrupt would prevent the decrypter from running, so you'd just be stepping through garbage. The way I worked around this was by writing a debugger that could hook the single step interrupt in such a way that it still forwarded the interrupt onto the previous hook. I still couldn't set breakpoints, but I could step through the code, watching it decode itself as it proceeded.
- caf 10y ago
  If you're single-stepping you don't need to patch the instruction stream for a breakpoint; you can just test the instruction pointer against the break address at each step.
  - Annatar 10y ago
    Not every processor has the means to perform direct comparisons against the program counter; as far as I know 6502 and MC68000 families can not do direct comparisons with the program counter.
    - caf 10y ago
      It's the program counter of the state that was interrupted by the single-step trap that's of interest, and that program counter is generally pushed onto the stack by the trap. For example, this is how the trace exception on the M68k works - the program counter of the next instruction to be executed can be read off the stack by the exception handler. The 6502 doesn't have built-in software single-stepping but the same effect was sometimes achieved by tying a short timer to the NMI - and when the NMI is asserted, the interrupted program counter is pushed onto the stack.
trungaczne 10y ago
Each process can be ptraced by only once process at any time, so if the malicious program want to avoid debuggers, they can attempt a ptrace() on their own process. If it fails, they know they are being debugged.
allenbrunson 10y ago
sometimes you don't have to bother detecting whether your program is being debugged, you can just adopt behavior that would make it impossible to run your code under a debugger. i got my start in programming by trying to break various copy protection schemes in MS-DOS games. i used a software debugger that would set int 3 breakpoints, as the article mentions. BUT, for the debugger to work, it has to rewrite the interrupt vector table so that int 3 instructions will cause the debugger's own code to be executed. really wily code would use the entry for int 3 in the interrupt vector table as a scratch variable to perform a bunch of its own calculations, perhaps thousands of them so you can't find and modify them all, which would ultimately wind up with an address to jump to, to continue execution of the program. this means that, when the debugger rewrites the int 3 vector, the program's internal calculations would be thwarted, effectively stopping the program at that point. this was really difficult to get past. there were other people doing this kind of thing who seemed to find ways around this technique, but i never did.
EvanAnderson 10y ago
It's essentially an arms-race. For every anti-debugging or obfuscation technique there's a reverse engineer who is willing to do hard work to overcome it. It becomes a kind of psychological warfare-- trying to wear-down the reverse enginner to make the give up. If this kind of thing interests you check out 4AM's Apple II cracks. He's removing the copy protection on old software so the techniques aren't directly applicable to current machines, but there's a font of creativity in the code he's reverse engineering. https://twitter.com/a2_4am https://twitter.com/a2_4am

by jtchang 10y ago

colejohnson66 10y ago

The most basic one asks the kernel if there's a debugger attached. Others check if certain programs are running. There's more complex ones, but I can't remember them right now

gst 10y ago

Lots of different approaches: To figure out if a process is ptraced the process can try to ptrace itself. An executable can only get ptraced once at a time, so the second call to ptrace will fail. Of course you can workaround that check by patching the code so that it doesn't call ptrace. Breakpoints are also easy to spot. Just look for 0xCC in the memory. But of course there are also multiple ways to work around those checks (such as using harder to detect hardware breakpoints instead of software breakpoints).

rollulus 10y ago

What I recall from the old days: it was indeed possible to detect Softice, through various ways, for instance simply trying to use its own API (through int3 as well having registers set to special magic values). Another trick to detect breakpoints as an anti-debugging measure was to compute a key over the block of memory that was to be protected. Any int3s in there would give a different output. A countermeasure was to use "memory breakpoints" which work via the x86 debug registers: https://en.wikipedia.org/wiki/X86_debug_register https://en.wikipedia.org/wiki/X86_debug_register

mappu 10y ago

An ideal debugger can't be detected. In the specific case of x86 and int3 bps, you can abuse variable-length instruction encoding to jump into the middle of another instruction. Then code execution can differ depending on the presence of 0xcc.

userbinator 10y ago
One common trick was to make the code itself part of a calculation for decrypting/checksumming/encryption/etc., so any modifications would cause a silent failure much later.
- crististm 10y ago
  The game King's Keep on Sinclair Spectrum was (in a way) like that. It had one or two obfuscated subroutines for xor encoding the main program. They were pretty well localized so once I identified them, decompiling the rest was trivial. Fun times then and now :)

pcwalton 10y ago

> How would a program detect the use of a debugger? IsDebuggerPresent() is the most straightforward way: https://msdn.microsoft.com/en-us/library/windows/desktop/ms680345(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

umanwizard 10y ago
The most straightforward way on one particular platform. There are definitely apps using anti-debugger techniques even on iOS.

dracht 10y ago

There's a bunch of ways. Some are obvious (like a call to IsDebuggerPresent()) and some not so much (like tripping some exception that only gets thrown when a process is not being debugged). Youtuber MalwareAnalysisForHedgehogs[0] has some interesting videos about this and anti-reversing tricks in general. https://www.youtube.com/channel/UCVFXrUwuWxNlm6UNZtBLJ-A https://www.youtube.com/channel/UCVFXrUwuWxNlm6UNZtBLJ-A

0x4a42 10y ago

There are many different techniques that you can find in this paper (wich, I think, is a reference on the subject): http://pferrie.host22.com/papers/antidebug.pdf http://pferrie.host22.com/papers/antidebug.pdf For a fresh compilation (which includes the previous paper) you can check this article: http://antukh.com/blog/2015/01/19/malware-techniques-cheat-sheet/ http://antukh.com/blog/2015/01/19/malware-techniques-cheat-s...

j4_james 10y ago

One technique I remember being used in DOS apps from many years ago was that the code would be encrypted in such a way that the next instruction to be executed would only be decrypted immediately before it was run. This was achieved by setting up the single step interrupt as the decrypter, and running the code in single step mode. The fact that the code was encrypted meant the debugger couldn't disassemble it in any meaningful way, and also made it impossible to set a breakpoint (since the breakpoint would just end up being "decrypted" into some other opcode that would inevitably crash). The debugger also couldn't step through the code, because taking over the single step interrupt would prevent the decrypter from running, so you'd just be stepping through garbage. The way I worked around this was by writing a debugger that could hook the single step interrupt in such a way that it still forwarded the interrupt onto the previous hook. I still couldn't set breakpoints, but I could step through the code, watching it decode itself as it proceeded.

caf 10y ago
If you're single-stepping you don't need to patch the instruction stream for a breakpoint; you can just test the instruction pointer against the break address at each step.
- Annatar 10y ago
  Not every processor has the means to perform direct comparisons against the program counter; as far as I know 6502 and MC68000 families can not do direct comparisons with the program counter.
  - caf 10y ago
    It's the program counter of the state that was interrupted by the single-step trap that's of interest, and that program counter is generally pushed onto the stack by the trap. For example, this is how the trace exception on the M68k works - the program counter of the next instruction to be executed can be read off the stack by the exception handler. The 6502 doesn't have built-in software single-stepping but the same effect was sometimes achieved by tying a short timer to the NMI - and when the NMI is asserted, the interrupted program counter is pushed onto the stack.

trungaczne 10y ago

Each process can be ptraced by only once process at any time, so if the malicious program want to avoid debuggers, they can attempt a ptrace() on their own process. If it fails, they know they are being debugged.

allenbrunson 10y ago

sometimes you don't have to bother detecting whether your program is being debugged, you can just adopt behavior that would make it impossible to run your code under a debugger. i got my start in programming by trying to break various copy protection schemes in MS-DOS games. i used a software debugger that would set int 3 breakpoints, as the article mentions. BUT, for the debugger to work, it has to rewrite the interrupt vector table so that int 3 instructions will cause the debugger's own code to be executed. really wily code would use the entry for int 3 in the interrupt vector table as a scratch variable to perform a bunch of its own calculations, perhaps thousands of them so you can't find and modify them all, which would ultimately wind up with an address to jump to, to continue execution of the program. this means that, when the debugger rewrites the int 3 vector, the program's internal calculations would be thwarted, effectively stopping the program at that point. this was really difficult to get past. there were other people doing this kind of thing who seemed to find ways around this technique, but i never did.

EvanAnderson 10y ago

It's essentially an arms-race. For every anti-debugging or obfuscation technique there's a reverse engineer who is willing to do hard work to overcome it. It becomes a kind of psychological warfare-- trying to wear-down the reverse enginner to make the give up. If this kind of thing interests you check out 4AM's Apple II cracks. He's removing the copy protection on old software so the techniques aren't directly applicable to current machines, but there's a font of creativity in the code he's reverse engineering. https://twitter.com/a2_4am https://twitter.com/a2_4am