Wi-Fi Direct UWP timeouts (Exception from HRESULT: 0x800705B4)

I'm starting a Wi-Fi Direct access point service using the UWP APIs. It starts OK. I'm using WiFiDirectConnectionListener to monitor for devices that get connected to the access point using the ConnectionRequested event.

var connectionRequest = args.GetConnectionRequest();
var deviceInformation = connectionRequest.DeviceInformation;

// FromIdAsync needs to be called from the UI thread (in MS example).
var isConnected = RunOnUIThreadAsync(() =>
{
    try
    {
        var device = WiFiDirectDevice.FromIdAsync(deviceInformation.Id).AsTask().Result;
        if (device != null)
        {
            device.ConnectionStatusChanged -= OnDeviceConnectionStatusChanged;
            device.ConnectionStatusChanged += OnDeviceConnectionStatusChanged;

            return true;
        }

        return false;
    }
    catch (Exception e)
    {
        // This throws an Exception from HRESULT: 0x800705B4.
        return false;
    }
}).Result; 

On some devices that get connected to the access point, an exception is thrown on calling FromIdAsync with

This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4).

In turn, the device that tries to the access point will not connect.

It's always the same devices that are unable to connect, while others connect just fine. I've tried with and without UI thread but the result remains the same. Am I using this wrong, or is this a bug in Wi-Fi Direct? If so, is there another way to start a Wi-Fi Direct access point without the UWP APIs? Perhaps that's working better.