Unhooking a hook set with SetWindowsHookEx more than once crashes the hooked process

-3

I am trying to hook the keyboard and mouse events of a target application. I followed the SO question How to hook external process with SetWindowsHookEx and WH_KEYBOARD, and the hooks are installed and uninstalled correctly the first time. However, after I uninstall the hook once and then install it again, trying to uninstall the hook the second time crashes the target application. The purpose of the hook is to monitor application idle time, so that I can do some tasks when the application is idle. I apologize for the length of the question, but I tried to put in all the details that might help Thanks

I need to be able to install and uninstall the hooks based on menu commands from a system tray icon. I have a console application [HookApp] which calls the install and uninstall methods in a DLL [HookDLL]. The console application also creates a window to handle menu events. I use the thread of the window to actually install and uninstall the hooks, because the same thread that installed the hook must uninstall it. The Console and the window both must be invisible. Only the system tray icon and its associated menu must be visible. I start the hooking application from the command line with the parameters i. ProcessId of the target process ii. Process Name iii. Idle time [in seconds] -Idle time before starting the action in the DLL I am wondering if the timer I start in the HookProcs is responsible for the crash.

Event Viewer Logs Faulting application name: notepad.exe, version: 10.0.17134.1, time stamp: 0x9d4727c2 Faulting module name: HookDLL_x64.dll_unloaded, version: 0.0.0.0, time stamp: 0x5c31aabd Exception code: 0xc0000005 Fault offset: 0x00000000000ba505 Faulting process id: 0x2bac

The exception code seems to suggest a memory access violation.

I have tried the following I. a. Call the SendMessageTimeout API with HWND_BROADCAST from HookApp b. Call the SendMessageTimeout API with HWND_BROADCAST from HookDLL based on Unloading DLL from all processes after unhooking global CBT hook

II. a. call FreeLibraryAndExitThread API in the DLLMain method on DLL_PROCESS_DETACH AND DLL_THREAD_DETACH -both together and singly based on http://www.rohitab.com/discuss/topic/42505-unloading-dll-crashes-exe/

III. Made the HookApp console and the window visible and hidden to see if it makes any difference

IV. Was unable to try the following because of 64 bit architecture https://www.unknowncheats.me/forum/programming-for-beginners/73377-unload-injected-dll-thread-process.html

Other Links I referred to a. Several programs crash when unhooking with UnhookWindowsHookEx() b. How to correctly use SetWindowsHookEx & CallNextHookEx c. Unloading an Injected DLL d. FreeLibraryAndExitThread crashes program when unloading injected DLL

//Code in the DLL
extern "C" __declspec(dllexport) void install(unsigned long threadID,int _nIdleTime) {
    nIdleTime = _nIdleTime;
    Log(L"install proc called from app: _nIdleTime", _nIdleTime);
    hhKeyboard = SetWindowsHookEx(WH_KEYBOARD, KeyboardProc, hinst, threadID);
    hhMouse = SetWindowsHookEx(WH_MOUSE, MouseProc, hinst, threadID);
    logger->Log("Install");
    logger->Log("Keyboard", (LONG)hhKeyboard);
    logger->Log("Mouse", (LONG)hhMouse);

}

//Uninstall the dll from the Target Process
DWORD WINAPI UnInjectDLLFromTarget() {
    uninstall();
    //DWORD_PTR dwResult = 0;
    //SendMessageTimeout(HWND_BROADCAST, WM_NULL, 0, 0, SMTO_ABORTIFHUNG | SMTO_NOTIMEOUTIFNOTHUNG, 1000, &dwResult);
    return TRUE;
}

LRESULT CALLBACK KeyboardProc(int code, WPARAM wParam, LPARAM lParam) {
    if (code < 0) {

        return CallNextHookEx(0, code, wParam, lParam);
    }
    StartTimer();
    Beep(1000, 20);


    return CallNextHookEx(hhKeyboard, code, wParam, lParam);
}

LRESULT CALLBACK MouseProc(int code, WPARAM wParam, LPARAM lParam) {
    if (code < 0) {

        return CallNextHookEx(0, code, wParam, lParam);
    }
    //  StartTimer();
        //Beep(1000, 20);

    return CallNextHookEx(hhMouse, code, wParam, lParam);
}
BOOL WINAPI DllMain(__in HINSTANCE hinstDLL, __in  DWORD fdwReason, __in  LPVOID lpvReserved) {


    if (fdwReason == DLL_PROCESS_DETACH || fdwReason==DLL_THREAD_DETACH) {
        FreeLibraryAndExitThread(hinstDLL, 0);
        return 0;
    }
}

//Code in the Application [HookApp]
Similar to  [But not exactly the same - It is a bit more involved because of the need to create the message pump and the menu event handling window]
 HINSTANCE hinst = LoadLibrary(_T("MyDLL.dll")); 

    if (hinst) {
        typedef void (*Install)(unsigned long);
        typedef void (*Uninstall)();

        Install install = (Install) GetProcAddress(hinst, "install");
        Uninstall uninstall = (Uninstall) GetProcAddress(hinst, "uninstall");

        install(threadID);

        Sleep(20000);

        uninstall();
    }
  1. SendMessageTimeout in the DLL [HookDLL] method Uninstall immediately crashes the application
  2. SendMessageTimeout in the application [HookApp] doesnt appear to do anything useful.
  3. FreeLibraryAndExitThread does not appear to do anything useful. However, with 2 and 3, I think I am able to install the hook for the second time. Without them, the first unhook crashes the application.
  4. I was unable to try the suggestion using assembly language inline because of 64 bit architecture.

Please help.

c++
winapi
64bit
setwindowshookex
unhookwindowshookex
asked on Stack Overflow Jan 6, 2019 by Sagar Kapadia

1 Answer

1

for uninstall hook and unload your dll all what you need - call UnhookWindowsHookEx for every hook handle obtained by a previous call to SetWindowsHookEx. all. you not need call FreeLibrary[AndExitThread] yourself. system auto call FreeLibrary on your hook dll after UnhookWindowsHookEx callend, when first (any) message will be received by target application thread and this thread call GetMessage or PeekMessage. so for just unload your dll after several UnhookWindowsHookEx - you need post message to thread, for which you install hook. if you do this for concrete single thread (dwThreadId) - you need PostThreadMessageW(dwThreadId, WM_NULL, 0, 0); - system yourself call FreeLibrary ( called from user32!__ClientFreeLibrary which called from ntdll!KiUserCallbackDispatcher - called inside GetMessage or PeekMessage when thread (or it window) got any message)

call FreeLibrary from dll for yourself - always error - if assume dll will be unloaded by this call - to where we return after call ? to unloaded place. call FreeLibraryAndExitThread exist sense in some cases, but only from thread, which you create yourself. call FreeLibraryAndExitThread from dll entry point at all fatal error - you kill not self thread, thread which you kill - hold critical section (loader lock) inside which called dll entry point - so fatal error. and at all absolute senseless call here - if you say receive DLL_PROCESS_DETACH this mean that dll already in unload process. DLL_THREAD_DETACH - arbitrary point, why and how many time you try call unload for self here ? exit thread while holding process wide critical section without release it. fatal error.

also because the same thread that installed the hook must uninstall it. - this is not true. another thread from also can do this.

also StartTimer(); in your code look like suspicious here. what this code is do ? are and where you cancel timer ?

answered on Stack Overflow Jan 6, 2019 by RbMm • edited Jan 6, 2019 by RbMm

User contributions licensed under CC BY-SA 3.0