I am working on an overlay component of a project in C#/DirectX.

This project does not use WinForms or WPF by design.

This overlay is hybrid, where I want to render custom UI components via DirectX to my window and pass through all input that is irrelevant to the underlying window/application (that I don't own). I basically want a transparent full-screen window that acts as an interactable layer on-top of another application, where I determine what input events I consume, and what events I let pass through.

This overlay creates an HWND via RegisterClassEx/CreateWindowEx WIN API calls. Mostly for testing purposes, I pass in a boolean variable to determine if I want to enable "passthrough" (basically adding WS_EX_TRANSPARENT to the creation flags). My DX code renders to the window via CreateSwapChainForHwnd.

Reference code snippet:

        // prepare WNDPROC-equivalent code for processing WM_* messages

        wndProc = windowProcedure;  
        RuntimeHelpers.PrepareDelegate(wndProc);
        wndProcPointer = Marshal.GetFunctionPointerForDelegate(wndProc);

        // prepare window class registration structure

        PInvoke.WNDCLASSEX wndClassEx = new PInvoke.WNDCLASSEX()
        {
            cbSize = PInvoke.WNDCLASSEX.Size(),
            style = 0,
            lpfnWndProc = wndProcPointer,
            cbClsExtra = 0,
            cbWndExtra = 0,
            hInstance = IntPtr.Zero,
            hIcon = IntPtr.Zero,
            hCursor = PInvoke.LoadCursor(IntPtr.Zero, (int)PInvoke.IDC_STANDARD_CURSORS.IDC_ARROW),
            hbrBackground = IntPtr.Zero,
            lpszMenuName = randomMenuName,
            lpszClassName = randomClassName,
            hIconSm = IntPtr.Zero
        };

        // Register window class via WINAPI 

        PInvoke.RegisterClassEx(ref wndClassEx);

        // Prepare window basic style flags (WS_*)

        WS style = (WS.WS_POPUP | WS.WS_VISIBLE);

        // Prepare window extended style flags (WS_EX_*)

        WSEx exStyle;

        if (_Topmost)
        {
            if (_AllowPassthrough)
                exStyle = (WSEx.WS_EX_TOPMOST | WSEx.WS_EX_TRANSPARENT | WSEx.WS_EX_LAYERED | WSEx.WS_EX_TOOLWINDOW | WSEx.WS_EX_NOACTIVATE); 
            else
                exStyle = (WSEx.WS_EX_TOPMOST | WSEx.WS_EX_LAYERED | WSEx.WS_EX_TOOLWINDOW | WSEx.WS_EX_NOACTIVATE); 
        }
        else
        {
            if (_AllowPassthrough)
                exStyle = (WSEx.WS_EX_TRANSPARENT | WSEx.WS_EX_LAYERED | WSEx.WS_EX_TOOLWINDOW | WSEx.WS_EX_NOACTIVATE);
            else
                exStyle = (WSEx.WS_EX_LAYERED | WSEx.WS_EX_TOOLWINDOW | WSEx.WS_EX_NOACTIVATE);
        }

        // Create window via WINAPI 

        _WindowHandle = PInvoke.CreateWindowEx(
            (uint)exStyle,
            randomClassName,
            randomWindowName,
            (uint)style, 
            _Position.X, _Position.Y,
            _Size.X, _Size.Y,
            IntPtr.Zero,
            IntPtr.Zero,
            IntPtr.Zero,
            IntPtr.Zero
            );

        // SetLayeredWindowAttributes is required to define transparency 

            // BOOL SetLayeredWindowAttributes( HWND hwnd, COLORREF crKey, BYTE bAlpha, DWORD dwFlags);
            // Flag: LWA_ALPHA      0x00000002    Use bAlpha to determine the opacity of the layered window.
            // Flag: LWA_COLORKEY   0x00000001    Use crKey as the transparency color. 

        PInvoke.SetLayeredWindowAttributes(_WindowHandle, 0, 255, 0x2);
        PInvoke.UpdateWindow(_WindowHandle);

The transparent, topmost DirectX 11 window is rendering fine and behaving appropriately from a visual perspective.

If I set my _AllowPassthrough to TRUE, all input gets passed down to the underlying window.

If I set my _AllowPassthrough to FALSE, no input gets passed down to the underlying window.

I am using the SetWindowsHookEx() API function to get low-level mouse and keyboard information, but it appears that when I call CallNextHookEx() after determining the input isn't relative to me, it never gets to the underlying window for processing (if _AllowPassthrough is false) or never gets blocked/consumed (if _AllowPassthrough is true).

Inside my Hook procedure, I either return (IntPtr)1; to block further processing, or return PInvoke.CallNextHookEx(hookHandle, nCode, wParam, lParam); to let someone else handle it.

Most of my research have only lead me to discussions or articles about fully "click-through" overlay windows; not windows that are partially click-through in certain regions.

I'm guessing it has to do with my implementation of "WS_EX_LAYERED" window styling.

Should I be looking into the "WS_EX_NOREDIRECTIONBITMAP" style option (for use with the Windows composition engine) instead of "WS_EX_LAYERED" (doing pixel hit tests based off of SetLayeredWindowAttributes)?

If it is of any consequence, I am using sharpDX as my DirectX wrapper library. My project is built via VS2017 and using the latest C# language specs on .NET Framework 4.7.1.

EDIT 3/19/2019:

So I've done some extensive trial-and-error on this subject and I've concluded that....

without WS_LAYERED, mouse input/hit tests/ etc. never fall through to the underlying window no matter what.

with WS_LAYERED, my window stops receiving any WM_NCHITTEST or other cursor related messages. I can get these from using the WH_MOUSE_LL hook (and associated LowLevelMouseProc() message handler), but consuming the event and returning a non-zero value (to "prevent the system from passing the message to the rest of the hook chain or the target window procedure") still allows other "non-mouse related" messages to pass through, so even trying to squelch MOUSE_MOVE by consuming it and using SetCursorPos() to move the cursor still causes the window underneath to get a notification of a hover or mouse over (I can see buttons highlighting and tooltips will pop up).

If I don't use SetCursorPos() the mouse will lock in place, since the necessary code to move the cursor isn't seeing the mouse movement. I'm unsure if SetCursorPos() causes a MOUSE_MOVE event, but I'd assume it doesn't since I'm consuming those events anyways (and preventing lockup using a "last == current" check.) I have no idea what other system events are being generated. Information on the low level nature of the input message queue is severely lacking online.

As for "non-alpha" colors, that doesn't seem to make any difference; any painted colors, regardless of opacity, still pass through mouse input.

I have tried the aforementioned WS_EX_NOREDIRECTIONBITMAP and refactored my DX rendering code to use the Composition Engine instead, using the well-cited MSDN article by Kenny Kerr as a reference, and this made no difference.

I'm really stumped on this one.