Can you PInvoke multibyte ANSI to a varargs? What am I doing wrong?

** MAJOR UPDATE ** I made a minor mistake but I'm still curious about exactly what is happening.

The function I am calling is actually "fooV", a function with this signature:

foo(const char *, const char *, EnumType, va_list)

This clears up the AccessViolationExceptions I was getting, but doesn't explain why params parameters work for every other .NET type except for strings that will have to be converted to multibyte ANSI characters. I'm going to get the developer of the DLL to expose the version that actually uses ... in the parameter list, any hints for PInvoking if va_list is the parameter?

** old post **

This is related to, but different from a recent question I asked.

I have to use PInvoke to call a C library function that has the following signature:

foo(const char *, const char *, EnumType, ...)

The function configures a resource in a way that varies by StructType; the case I'm interested in configures something that expects the varargs to be a single ANSI multibyte string. I used this PInvoke signature to call the function:

    [DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
    public static extern int Foo(string s1, string s2, EnumType st1, params string[] s3);

The params string[] seemed odd to me, but previous signatures that called this function were using it for other types such as bool so I followed the pattern.

I wrap this with a friendlier .NET method:

public void Foo(string s1, string s2, string s3)
{
    int error = Dll.Foo(s1, s2, EnumType.Foo, s3);
    // handle errors
}

Recently I changed the signature to include "BestFitMapping = false, ThrowOnUnmappableChar = true" in the DLLImport attribute to comply with FxCop's suggestions. This is a red herring as I will describe later.

What I expect out of making this change is limited support for Unicode. For example, on a machine with an English code page an exception will be thrown if you pass a string that contains a Japanese character. On a Japanese machine, I'd expect to be able to pass a Japanese string to the function. The English test worked as expected, but the Japanese test throws a System.Runtime.InteropServices.COMException with HRESULT 0x8007007A. This happens even without the BestFitMapping and ThrowOnUnmappableChar settings.

I've done a little looking around and saw some sites that suggested you could PInvoke varargs by just specifying normal arguments, so I tried this signature:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
public static extern int Foo(string s1, string s2, EnumType st1, string s3);

This throws an AccessViolationException when I use it on either the English or Japanese machine.

I cannot use UTF-8 or another Unicode encoding because this C library expects and handles only multibyte ANSI. I have found that specifying CharSet.Unicode for this function works, but I'm worried it's only by coincidence and not something upon which I should rely. I have thought about converting the string to a byte[] using the system code page, but can't figure out how to specify I'd like to pass a byte array to the varargs parameter.

What's going on? On the English machine, English characters work fine and Japanese charactes throw an ArgumentException as expected. On the Japanese machine, English characters work fine and Japanese characters throw the COMException. Is there something wrong with my PInvoke signature? I've tried using the MarshalAs attribute to specify a type of LPArray and subtype of LPStr, but this fails in the same way. UnmanagedType.LPStr indicates it is a single-byte ANSI string; is there a way to specify a multibyte ANSI string?

** UPDATE ** Here's an addition of stuff taking current comments into account.

If I specify a calling convention of CDecl and take normal parameters like this:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo", CallingConvention = CallingConvention.Cdecl)]
public static extern int Foo(string s1, string s2, EnumType e1, string s3) 

When I use this specification, I get an AccessViolationException. So I tried this:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo", CallingConvention = CallingConvention.CDecl)]
public static extern int Foo(string s1, string s2, EnumType e1, string s3) 

This works for English and throws the COMException when I use Japanese characters.

The only thing I've found that works consistently is this:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo", BestFitMapping = false, ThrowOnUnmappableChar = true)]
public static extern int Foo(string s1, string s2, EnumType e1, params IntPtr[] s3)

To make this work, I use Marshal.StringToHGlobalAnsi() and pass that pointer. This works for English and Japanese. I'm not comfortable with not understanding why this is the solution, but it works.

You probably also need to specify CallingConvention=CallingConvention.Cdecl since a varargs function will use a _cdecl calling convention (the default is Winapi which in turn defaults to StdCall).

You may need something else, but I'm pretty sure you'll need at least that.