Why does CreateWindowEx intermittently fail in Fall Creators Update (error 998 / 0x3e6)?

After updating Windows 10 to the Fall Creators Update, I'm experiencing intermittent failures in CreateWindowEx. GetLastError returns 0x000003e6 -- ERROR_NOACCESS / invalid access to memory. Frequently (though not always), simply repeating the call with the exact same parameters succeeds.

hwnd = ::CreateWindowEx(0, wndClassName, NULL, WS_VISIBLE | WS_CHILD | WS_CLIPCHILDREN | WS_CLIPSIBLINGS,   r.left, r.top, r.right, r.bottom, hParent, NULL, AfxGetInstanceHandle(), NULL);

wndClassName is name of wndClass registered with this call:

LPCSTR wndClassName = /* clsName passed in */;
WNDCLASS wndcls;
HINSTANCE hInst = AfxGetInstanceHandle();
if (!(::GetClassInfo(hInst, wndClassName, &wndcls)))
{
    ZeroMemory(&wndcls, sizeof(wndcls));
    wndcls.lpfnWndProc = ::DefWindowProc;
    wndcls.hCursor = ::LoadCursor(nullptr, IDC_ARROW);
    wndcls.hInstance = hInst;
    wndcls.lpszClassName = wndClassName;
    wndcls.hbrBackground = (HBRUSH)::GetStockObject(NULL_BRUSH);
    AfxRegisterClass(&wndcls);
}

MSDN documentation states that one of the typical reasons for failure is

The WH_CBT hook is installed and returns a failure code

My application does use MFC and I believe MFC relies on a CBT hook.

Any ideas on how to investigate and resolve these persistent, intermittent failures that started occurring with Fall Creators Update?

This was a Windows 10 Fall Creators Update bug that was fixed in Win10 FCU Update KB4054517 released December 12.

The bug affected 32-bit large address aware applications running on 64-bit versions of Windows 10 Fall Creators Update. CreateWindowEx failed when enough memory below 0x80000000 was not available to the kernel.

For documentary purposes, these are the actions we took to get insight into the problem:

1. open a technical support request at http://support.microsoft.com/oas
2. get shuffled around until ending up in engineering technical support
3. have case assigned to the author of a blog post about troubleshooting CreateWindowEx failures
4. spend days trying to produce steps to repro the failures so that the problem can be inspected in the kernel debugger
5. fail to get case that can repro
6. another customer was able to send in a repro case that has similar CreateWindowEx failures with same error code (problem with 32-bit application running on 64-bit version of Win10 Fall Creators Update).
7. cross fingers and wait for possible fix in December Patch Tuesday