VB.Net calling win32 api on 32 and 64 bit systems, runs OK on 32 but not 64

To make an existing managed application with unmanaged 'appendages' 64-bit capable, I decided to rewrite a 32 bit unmanaged VC++ dll as a managed VB.Net class library.

The application must be able to run on any framework from 2.0 up, and the VC++ code I ported was using named pipes, which are supported only from framework 3.5 up. So I had to resort to reflection, calling windows API functions for everything related to the pipes.

The result works perfectly in 32 bit mode, but in 64 bit, I get all kinds of trouble. The actual behavior may change dramatically with even the least change in source code, just swapping some variable declarations around or moving a simple assignment statement to another location in the code may cause any effect from

• "a barely noticeable hiccup every X hours of running fine",
• "appear to receive some bad data from time to time" (it actually took WireShark and a decent portion of luck to find out that the first byte of a 13-byte packet received on the named pipe was passed as 0x00 to managed code, while it was actually received as 0x02 in the pipe data packet, while the rest of the packet was passed intact)

to

• "crash the entire .Net runtime with an error 0xc0000409 in mscorwks.dll soon as a client tries to connect to the pipe".

If you ask me, the quick change in symptoms, but especially the latter, points towards something somewhere, probably a pointer or handle, that is still being stored or passed to/from Win32 API at 32 bit sizes when running at 64 bit. But I don't succeed in figuring out what.

The most likely location is in the API declarations. The rest is all managed code, and compiled with options 'strict' and 'explicit' both on. Can someone have a look at the snippets below and see if he can spot something I keep overlooking?

Operation is asynchonous. The actual waits are performed at managed level on an array containing managed as well as unmanaged handles (through SafeWaitHandle wrapping), so events in managed together with unmanaged stuff can be dealt with by a single WaitHandle.WaitAny(...array...) call.

<DllImport("kernel32", SetLastError:=True)> Private Shared Function CreateNamedPipe(
    lpName As String,
    dwOpenMode As Int32,
    dwPipeMode As Int32,
    nMaxInstances As Int32,
    nOutBufferSize As Int32,
    nInBufferSize As Int32,
    nDefaultTimeOut As Int32,
    lpSecurityAttributes As IntPtr    ' when declared as SECURITY_ATTRIBUTES runtime won't accept passing Nothing, even when marked <[Optional]>
) As Microsoft.Win32.SafeHandles.SafeFileHandle : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function ConnectNamedPipe(
    hNamedPipe As SafeHandle,
    ByRef lpOverlapped As System.Threading.NativeOverlapped
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function DisconnectNamedPipe(
    ByVal hNamedPipe As SafeHandle
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function ReadFile(
    <[In]> hFile As SafeHandle,
    <Out> lpBuffer As IntPtr,
    <[In]> nNumberOfBytesToRead As Int32,
    <Out, [Optional]> ByRef lpNumberOfBytesRead As Int32,
    <[In], Out, [Optional]> ByRef lpOverlapped As System.Threading.NativeOverlapped
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function WriteFile(
    <[In]> hFile As SafeHandle,
    <[In]> lpBuffer As IntPtr,
    <[In]> nNumberOfBytesToWrite As Int32,
    <[Out], [Optional]> ByRef lpNumberOfBytesWritten As Int32,
    <[In], [Out], [Optional]> ByRef lpOverlapped As NativeOverlapped
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function CloseHandle(hHandle As SafeHandle) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function CloseHandle(hHandle As IntPtr) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function GetOverlappedResult(
    ByVal hFile As SafeHandle,
    ByRef lpOverlapped As System.Threading.NativeOverlapped,
    ByRef lpNumberOfBytesTransferred As Int32,
    ByVal bWait As Boolean
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function CancelIo(
    <[In]> hFile As SafeHandle
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function PeekNamedPipe(
    <[In]> hNamedPipe As SafeHandle,
    <Out, [Optional]> ByRef lpBuffer As Byte(),
    <[In]> nBufferSize As Integer,
    <Out, [Optional]> ByRef lpBytesRead As Integer,
    <Out, [Optional]> ByRef lpTotalBytesAvail As Integer,
    <Out, [Optional]> ByRef lpBytesLeftThisMessage As Integer
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function GetProcessHeap() As IntPtr : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function HeapAlloc(
    <[In]> hHeap As IntPtr,
    <[In]> dwFlags As Int32,
    <[In]> dwBytes As IntPtr
) As IntPtr : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function HeapFree(
    <[In]> hHeap As IntPtr,
    <[In]> dwFlags As Int32,
    <[In]> lpMem As IntPtr
) As Boolean : End Function

<DllImport("kernel32", SetLastError:=True)> Private Shared Function CreateEvent(
    <[In], [Optional]> lpEventAttributes As IntPtr,
    <[In]> bManualReset As Boolean,
    <[In]> bInitialState As Boolean,
    <[In], [Optional]> lpName As String
) As IntPtr : End Function

This is how CreateEvent is used to create a handle managed code can wait for:

Private Shared Function AllocateEventHandle() As Microsoft.Win32.SafeHandles.SafeWaitHandle
    Return New Microsoft.Win32.SafeHandles.SafeWaitHandle(CreateEvent(Nothing, False, False, Nothing), True)
End Function

Buffers to be used in calls to unmanaged code are created like this:

Private Shared Function AllocateBuffer(nBytes As Integer) As IntPtr
    Return HeapAlloc(GetProcessHeap(), 0, New IntPtr(nBytes))
End Function