Using NtCreateMutant in usermode

I tried creating a mutant in my process that closes the process when it is created more than once with this code:

HANDLE m;

OBJECT_ATTRIBUTES attr;
attr.Length = sizeof(OBJECT_ATTRIBUTES);

UNICODE_STRING str;
RtlInitUnicodeString(&str, L"123");

attr.ObjectName = &str;

long stats;

if (!(stats = NtCreateMutant(&m, MUTANT_ALL_ACCESS, &attr, false))) {
    if (0xC0000035 == stats) {
        return false;
    }

    return true;
}

All of the functions have the correct address

When I remove the code attr.ObjectName = &str; the mutant handle is not null, but the second process does not exit since the NTSTATUS is 0

When I keep the code, NTSTATUS returns STATUS_OBJECT_PATH_SYNTAX_BAD, which I have no idea what has to do with NtCreateMutant

I want to know if it is even possible to do this using NtCreateMutex in usermode and to fix these issues or if I should just stop trying and just use CreateMutex (I dont want to use it for no reason other than I wanted to try using NtCreateMutex)

I know it's 1 year late but considering how many questions are on stack overflow, better later than never.

I`m quite surprised why @Raymond Chen made that comment, because you can use undocumented API's just fine. They are everywhere in Windows, and they hardly changed for the last decade.

They are also used in literally every Windows kernel driver possible, so they are not really "undocumented" and "unsupported".

First of all, that's not how you initialise OBJECT_ATTRIBUTES; if you want to do it that way you would do:

OBJECT_ATTRIBUTES ObjectAttributes; = { sizeof(ObjectAttributes) };

The recommended and official way is to use the InitializeObjectAttributes macro.

Second, the mutex name cannot be "123" inside the NT namespace. That's why you get STATUS_OBJECT_PATH_SYNTAX_BAD.

Third, your code is just ugly written and the usage is bad. What you want to create (from what it sounds) is a "process barrier".

I will give you a working full sample code:

BOOL CreateProcessBarrier()
{
    NTSTATUS Result;
    UNICODE_STRING MutantName;
    OBJECT_ATTRIBUTES ObjectAttributes;
    HANDLE MutantHandle;
    WCHAR MutantNameNT[] = L"\\??\\40691290-71d5-45bc-b86a-e714496f4bf2";
    WCHAR MutantNameWin[] = L"40691290-71d5-45bc-b86a-e714496f4bf2";

    RtlInitUnicodeString(&MutantName, MutantNameNT);

    InitializeObjectAttributes(&ObjectAttributes, &MutantName, 0, NULL, NULL);

    // ALL OK
    if (0 <= (Result = ZwCreateMutant(&MutantHandle, MUTANT_ALL_ACCESS, &ObjectAttributes, TRUE)))
    {
        // Banana
        return TRUE;
    }
    // Already exists
    else if (Result == STATUS_OBJECT_NAME_COLLISION || Result == STATUS_OBJECT_NAME_EXISTS)
    {
        // Que paso ?
        return FALSE;
    }
    // Mutex creation failed, fallback to winapi
    else
    {
        // Papaya
        CreateMutexW(NULL, TRUE, MutantNameWin);

        if (GetLastError() == ERROR_ALREADY_EXISTS)
        {
            return FALSE;
        }
    }

    return TRUE;
}

The function above has support to fallback to normal WinApi in case it fails.

Usage is simple as:

Process A:

CreateProcessBarrier();

Process B,C,D,etc:

if(!CreateProcessBarrier()) ... do whatever.

Enjoy.