Malware reads memory at specific addresses to check if the Windows API functions are hooked.
This method is based on the fact, that emulation environments are most likely to hook these functions to be able to gather data and statistics during emulation.
Popular functions to be checked:
Reading memory is accomplished via the following functions:
Then different algorithms may be used for checking:
Comparing first two bytes with \x8B\xFF (mov edi, edi) — typical prologue start for kernel32 functions.
Comparing first N bytes with \xCC - software breakpoint (int 3), not connected with hooks directly but still a suspicious behavior.
Comparing first N bytes with \xE9 (call) or with \xEB (jmp instruction) — typical instructions for redirecting execution.
Checking for push/ret combo for execution redirection.
and so on.
It’s pretty tricky to count for every possible comparison so general indication of something unusual in application’s behavior is reading memory where OS libraries reside. If to be more precise: reading memory where “interesting” functions are situated.
This atricle explains how to detect user-mode hooks and remove them. The following code samples are taken from the article.
Malware sets mouse hook to detect a click (or more) if it occurs. If it’s the case malware treats the host a usual one, i.e., with end user behind the screen - not a virtual environment. If no mouse click is detected then it’s very likely a virtual environment.
Though Check Point tool InviZzzible has them all implemented, due to modular structure of the code it would require more space to show a code sample from this tool for the same purposes. That’s why we’ve decided to use other great open-source projects for examples throughout the encyclopedia.