Why would any DLL being injected crash a host process?

2

In beta testing a new release of my software, several users reported exceptions when running the app. In both cases it's: "The application was unable to start correctly (0xc0000142)". I've also seen it as 0xc0000005. I found a local system with this error as well and found when running it under a debugger, "datamngr.dll" had an access violation and failed allocating on the heap. I quickly discovered "datamngr.dll" is spyware and is being loaded as it was in the system's AppInit.

Once I cleared the AppInit reg key, this problem went away. I checked it out via Process Monitor, and any time this DLL was being injected, my application crashed. I thought it was just badly written spyware, but I've since found other DLLs doing the same thing (such as acaptuser32.dll, which is legitimate software). What's odd to me is the previous version of my software does not crash. There's been many, many changes between the two versions so it's hard to say what it is.

Where do I start here? Some online exploring shows apps like Firefox replace LoadLibrary to blacklist DLLs from being injected. But I'd like to start from the more basic -- why is the application now crashing when it didn't before?

I realize this is very vague, but that's pretty much unavoidable. I'm hoping there's something obvious in the properties for the project I'm doing incorrectly. I've tried with ASLR on and off, DEP on and off...I've tried delay-loading user32.dll and manually loading it via LoadLibrary (with SetErrorMode set to ignore errors), and nothing is working for me. We've seen this happen on Windows XP and Windows 7 (32 and 64-bit).

Any pointers on where to start would be greatly appreciated. I'll provide as much information as I can if anyone needs other details.

Cheers

windows
access-violation
dll-injection
asked on Stack Overflow Feb 7, 2012 by SentientAI

1 Answer

1

I did find a fix. I used Process Monitor to compare the order of DLL loading in versions with and without DLL injectors. One thing that then struck me is the C++ DLL I have which loads .NET (via LoadLibrary) was being included first. Because the CLR is such a large beast, I decided to try delay-loading that DLL and all .NET DLLs. That's all it took - my problem has gone away.

So it's as Raymond Chen said -- the ordering is fragile. If other people encounter this issue, I suggest just tweaking your DLL loading order.

answered on Stack Overflow Feb 7, 2012 by SentientAI

User contributions licensed under CC BY-SA 3.0