ACCESS_VIOLATION calling Btrieve BTRCALL function from Rust

I am attempting to call Btrieve (a very old database engine) from Rust. This is a bit long, but this is my first attempt at FFI from Rust and I wanted to describe everything I have done.

The Btrieve engine is implemented in a DLL, w3btrv7.dll, which is a 32-bit DLL. I have made an import library for it using 32-bit MSVC tools (it doesn't come with an official one):

lib /Def:w3btrv7.def /Out:w3btrv7.lib /Machine:x86

I then installed the 32-bit Rust toolchain stable-i686-pc-windows-msvc and set it as my default. Bindgen barfs on the official Btrieve headers so I had to make my own. Luckily we only need to wrap a single function, BTRCALL.

I have this in my wrapper.h:

short int BTRCALL(
    unsigned short  operation,
    void*           posBlock,
    void*           dataBuffer,
    unsigned short* dataLength,
    void*           keyBuffer,
    unsigned char   keyLength,
    char            ckeynum);

I am linking as:

println!("cargo:rustc-link-lib=./src/pervasive/w3btrv7");

Which seems to work: the program runs, is a 32-bit exe, and I can see in Process Explorer that it has loaded w3btrv7.dll.

When I send the header through bindgen I get:

extern "C" {
    pub fn BTRCALL(
        operation: ::std::os::raw::c_ushort,
        posBlock: *mut ::std::os::raw::c_void,
        dataBuffer: *mut ::std::os::raw::c_void,
        dataLength: *mut ::std::os::raw::c_ushort,
        keyBuffer: *mut ::std::os::raw::c_void,
        keyLength: ::std::os::raw::c_uchar,
        ckeynum: ::std::os::raw::c_char,
    ) -> ::std::os::raw::c_short;
}

The types and sizes all seem to tally up correctly, and they match a DllImport I have from a C# application which works perfectly:

[DllImport("w3btrv7.dll", CharSet = CharSet.Ansi)]
private static extern short BTRCALL(
    ushort operation, // In C#, ushort = UInt16.
    [MarshalAs(UnmanagedType.LPArray, SizeConst = 128)] byte[] posBlock,
    [MarshalAs(UnmanagedType.LPArray)] byte[] dataBuffer,
    ref ushort dataLength,
    [MarshalAs(UnmanagedType.LPArray)] byte[] keyBuffer,
    byte keyLength,     // unsigned byte
    char keyNumber);    // 2 byte char

The keyNumber is slightly different, but I have tried both bytes and shorts in both signed and unsigned variations, and it still doesn't work.

Unfortunately when I run my program it blows up after the first call to BTRCALL. (Well, actually it's when the function that this call is in returns). I've extracted all the params into local variables and checked their types and all looks correct:

let op: u16 = 0;
let mut pos_block: [u8; 128] = self.pos_block.clone();
let pos_block_ptr: *mut std::ffi::c_void = pos_block.as_mut_ptr() as *mut _;
let mut data_buffer: [u8; 32768] = self.data_buffer.clone();
let data_buffer_ptr: *mut std::ffi::c_void = data_buffer.as_mut_ptr() as *mut _;
let mut data_length: u16 = data_buffer.len() as u16;
let mut key_buffer: [u8; 256] = self.key_buffer.clone();
let key_buffer_ptr: *mut std::ffi::c_void = key_buffer.as_mut_ptr() as *mut _;
let key_length: u8 = 255; //self.key_length;
let key_number: i8 = self.key_number.try_into().unwrap();

let status: i16 = BTRCALL(
    op,
    pos_block_ptr,
    data_buffer_ptr,
    &mut data_length,
    key_buffer_ptr,
    key_length,
    key_number
);

It crashes the program with

error: process didn't exit successfully: `target\debug\blah.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

From what I have read, this is probably due to an improper address access.

Indeed, when I put some tracing in to check the variables there is some very interesting behaviour, in that my local variables which are passed by value seem to be getting overwritten. The log here just dumps the first 30 bytes of the buffers because the rest is just zeros:

pos_block = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
pos_block_ptr = 0xad6524
data_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
data_buffer_ptr = 0xad65a8
data_length = 32768
key_buffer = [34, 67, 58, 92, 116, 101, 109, 112, 92, 99, 115, 115, 92, 120, 100, 98, 92, 67, 65, 83, 69, 46, 68, 66, 34, 0, 0, 0, 0, 0]
key_buffer_ptr = 0xade5b0
key_length = 255
key_number = 0

>>>>>>>>>>>>>>> AFTER THE CALL TO BTRCALL:

pos_block = [0, 0, 0, 0, 0, 0, 0, 0, 0, 76, 203, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0]
pos_block_ptr = 0x0
data_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
data_buffer_ptr = 0x42442e45
data_length = 0
key_buffer = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
key_buffer_ptr = 0x0
key_length = 173
key_number = 0
BTRCALL() returned B_NO_ERROR

Notice pos_block_ptr has been set to 0, among other things. In contrast, a successful execution of the exact same call from the C# code simply writes some data into the first 18 bytes of pos_block and doesn't change any of the other variables.

It's as if it went a bit berserk and just started overwriting memory...

At this point I don't know what to try next.

Changing the declaration from extern "C" to extern "stdcall" works:

extern "stdcall" {
    pub fn BTRCALL(
        operation: ::std::os::raw::c_ushort,
        posBlock: *mut ::std::os::raw::c_void,
        dataBuffer: *mut ::std::os::raw::c_void,
        dataLength: *mut ::std::os::raw::c_ushort,
        keyBuffer: *mut ::std::os::raw::c_void,
        keyLength: ::std::os::raw::c_uchar,
        ckeynum: ::std::os::raw::c_char,
    ) -> ::std::os::raw::c_short;
}