Custom Lua allocation function's freeing throws COMException

I'm trying to use a custom allocation function to control the amount of memory used by Lua, but for some reason I get a COMException while Lua is trying to free memory, saying that an argument is invalid (HRESULT: 0x80000003), if I let a script that creates and extends strings run for a few seconds. Here's my custom allocation function:

private static IntPtr Alloc(IntPtr ud, IntPtr ptr, int osize, int nsize)
{
    if (nsize == 0)
    {
        Marshal.FreeHGlobal(ptr);
        return IntPtr.Zero;
    }
    else
    {
        if (osize == 0)
            return Marshal.AllocHGlobal(nsize);
        else
            return Marshal.ReAllocHGlobal(ptr, (IntPtr)nsize);
    }
}

There's nothing special going on, I simply set that function with lua_setallocf and then start to create and concat strings in a loop, to cause memory changes. The same thing happens in managed C++, using realloc and free, though I don't know if those aren't actually doing the same exact thing behind the scenes.

If I use Lua's stock allocation function everything works fine. Does anyone have an idea what might be causing this?

Update: I just discovered that this only happens using LuaJit, but not with the official Lua 5.1 binaries...