How to create truly connectionless UdpClient?

My goal is to create UDP listener that handles all datagrams incoming to specified port, regardless of its origin. Then it responds to that origin with some message.

public void StartListening(int port = 13000)
{
    ListenerPort = port;
    udpClient = new UdpClient(ListenerPort);

    udpClient.BeginReceive(new AsyncCallback(handleIncomingMessages), null);
}

private void handleIncomingMessages(IAsyncResult ar)
{
    var receivedData = udpClient.EndReceive(ar, ref senderIpEndPoint);
    //[...]
    udpClient.BeginReceive(new AsyncCallback(handleIncomingMessages), null);
}

Everything worked fine until I turned off one of those "origins". After a few seconds my listener caught SocketException.

An existing connection was forcibly closed by the remote host.

I always thought UDP is a connectionless protocol, yet it seems UdpClient tracks somehow that remote host is available. Is there some standard, .NET Core way I could disable this?

EDIT:

Here is error stack trace:

System.Net.Sockets.SocketException
  HResult=0x80004005
  Message=An existing connection was forcibly closed by the remote host.
  Source=System.Net.Sockets
  StackTrace:
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.EndReceiveFrom(IAsyncResult asyncResult, EndPoint& endPoint)
   at System.Net.Sockets.UdpClient.EndReceive(IAsyncResult asyncResult, IPEndPoint& remoteEP)
   at NetworkController.UDP.NetworkManager.handleIncomingMessages(IAsyncResult ar) in [...]
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.ContextAwareResult.CompleteCallback()
   at System.Net.ContextAwareResult.<>c.<Complete>b__15_0(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)