Occasional drop of connection with Socket exception 0x80004005

I'm experiencing a problem of randomly getting cryptic errors when receiving data with the TCP server that I've written. The exception is the following:

System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at TTS_Simulator.TCPChannel.ReceiveData(IAsyncResult ar)

I'm using BeginXXX and EndXXX asynchronous methods and the other side definitely did not close the connection.

The piece of code responsible for accepting the connection:

    protected void AcceptCallback(IAsyncResult ar)
    {
        try
        {
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);
            _logger.Info("Connection request accepted on local endpoint: " + listener.LocalEndPoint);
            _logger.Info("Connection request accepted from remote endpoint: " + handler.RemoteEndPoint);
            Thread.Sleep(100);

            StateObject state = new StateObject { workSocket = handler };
            handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                ReceiveData, state);
            IsConnected = true;
            ControlActivator?.Invoke(true);
            listener.BeginAccept(AcceptCallback, listener);
        }
        catch (SocketException ex)
        {
            _logger.Error(ex);
        }
        catch (NullReferenceException ex)
        {
            _logger.Error(ex);
        }
    }

And the piece of code responsible for receiving the messages:

    protected void ReceiveData(IAsyncResult ar)
    {
        StateObject state = (StateObject)ar.AsyncState;
        Socket handler = state.workSocket;
        string _inputData = String.Empty;
        int totalBytesRead = 0;
        int bytesRead = 0;

        try
        {
            bytesRead = handler.EndReceive(ar);
            totalBytesRead += bytesRead;

            if (totalBytesRead == 0)
            {
                _logger.Info("Client disconnected at: " + handler.LocalEndPoint);
                handler.Disconnect(true);
                handler.Close();
                handler.Dispose();
                IsConnected = false;
                ControlActivator?.Invoke(false);
                return;
            }

            if (totalBytesRead > 0)
            {
                if (handler.Available > 0)
                {
                    bytesRead = handler.Receive(state.buffer, bytesRead, StateObject.BufferSize,
                        SocketFlags.None);
                    totalBytesRead += bytesRead;
                }
            }
            DataHandler(handler, state.buffer, totalBytesRead);
            totalBytesRead = 0;
            state.workSocket.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, ReceiveData, state);
        }
        catch (System.Net.Sockets.SocketException ex)
        {
            _logger.Error("Error receiving data! \n" + ex);
        }
        catch (Exception ex)
        {
            _logger.Error("Error when processing the data! \n" + ex);
        }
    }

Sometimes it happens that both the server and the client report the exactly same problem, and sometimes, it's just either the server or the client. What is also interesting, is that when the server accepts the connection, I can see in the logs that there are actually two connection requests, both being accepted. Don't know if that has anything to do with it... Just a side note, both the server and the client run on the same machine, which is an MS Azure VM. If need be, I can also attach the entire code for the server.

Thanks!