Hooks detection methods
1. Check whether hooks are set within system functions
2. Check user clicks via mouse hooks
3. Check for incorrectly hooked functions
Techniques described here make use of hooks either to detect user presence or as means to be checked whether some unusual-for-host-OS hooks installed.
Malware reads memory at specific addresses to check if Windows API functions are hooked.
This method is based on the fact, that emulation environments are most likely to hook these functions to be able to gather data and statistics during an emulation.
Popular functions to be checked:
Reading memory is accomplished via the following functions:
Then different algorithms may be used for checking:
- Comparing first two bytes with \x8B\xFF (mov edi, edi) — typical prologue start for kernel32 functions.
- Comparing first N bytes with \xCC - software breakpoint (int 3), not connected with hooks directly but still a suspicious behavior.
- Comparing first N bytes with \xE9 (call) or with \xEB (jmp instruction) — typical instructions for redirecting execution.
- Checking for push/ret combo for execution redirection.
and so on.
It’s pretty tricky to count for every possible comparison so general indication of something unusual in application’s behavior is reading memory where OS libraries reside. If to be more precise: reading memory where “interesting” functions are situated.
This atricle explains how to detect user-mode hooks and remove them. The following code samples are taken from the article.
Example of hook detection
Example of unhooking functions
This technique is described by this link (p.4, p.7).
Malware sets mouse hook to detect a click (or more) if it occurs. If it’s the case malware treats the host a usual one, i.e., with end user behind the screen - not a virtual environment. If no mouse click is detected then it’s very likely a virtual environment.
- SetWindowsHookExA/W (WH_MOUSE_LL, ...)
Code sample (SetWindowsHookExA)
Code sample (GetAsyncKeyState)
There are more than 400 Native API functions (or Nt-functions) in ntdll.dll that are usually hooked in sandboxes.
In such a large list, there is enough space for different kinds of mistakes. We checked the hooked Nt-functions in popular sandboxes
and found several issues. On of them is a lack of necessary checks for arguments in a hooked function. This case is
described our article “Timing: Call a potentially hooked delay function with invalid arguments evasions”
Another issue we found is a discrepancy in the number of arguments in a hooked and an original function. If a function is hooked incorrectly, in kernel mode this may lead an operating system to crash. Incorrect user-mode hooks are not as critical. However, they may lead an analyzed application to crash or can be easily detected. For example, let’s look at the NtLoadKeyEx function. It was first introduced in Windows Server 2003 and had only 4 arguments. Starting from Windows Vista up to the latest version of Windows 10, it has 8 arguments:
; Exported entry 318. NtLoadKeyEx ; Exported entry 1450. ZwLoadKeyEx ; __stdcall NtLoadKeyEx(x, x, x, x, x, x, x, x) public _NtLoadKeyEx@32
However, in the Cuckoo monitor, the NtLoadKeyEx declaration still has only 4 arguments:
* POBJECT_ATTRIBUTES TargetKey * POBJECT_ATTRIBUTES SourceFile ** ULONG Flags flags ** HANDLE TrustClassKey trust_class_key
We found this legacy prototype used in other sources as well. For example, CAPE monitor has the same issue:
Therefore, if a sandbox uses any recent Windows OS, this function is hooked incorrectly. After the call to the incorrectly hooked function, the stack pointer value becomes invalid. Therefore, a totally “legitimate” call to the RegLoadAppKeyW function, which calls NtLoadKeyEx, leads to an exception. This fact can be used to evade Cuckoo and CAPE sandbox with just a single call to the RegLoadAppKeyW function.
Instead of using RegLoadAppKeyW, we can call the NtLoadKeyEx function directly and check the ESP value after the call.
No signature recommendations are provided for this evasion group as it’s hard to make a difference between the code which aims for some evasion technique and the one which is “legally used”.
- versus function hook checks: set kernel mode hooks; second solution is to use stack routing to implement function hooking;
- versus mouse click checks via hooks: use mouse movement emulation module.
- versus incorrect function hooks: ensure all the hooked function have the same number of arguments as the original functions
Credits go to user dtm from 0x00sec.org forum.
Due to modular code structure of the Check Point’s tool called InviZzzible it would require more space to show a code sample from this tool for the same purposes. That’s why we’ve decided to use other great open-source projects for examples throughout the encyclopedia.