Windows Service terminated unexpectedly while executing EndAcceptTcpClient - i.e while accepting new socket client

0

I have a Windows Service (Server) which is listening on a port 25101. The clients can connect to the server via this port. This is a secure port and therefore client and server requires the TLS handshake once the TCP/IP/Socket level connection is established.

The Windows Service is written in the .NET framework using the C#.

The abstract of the source code is written below.


Server Code


TcpClient client = null;

try
{
// This is where the service process is terminating  
client = m_listener.EndAcceptTcpClient(ar); 
}
catch (Exception ex)
{
  s_log.Error(ex, "Exception EndAcceptTcpClient");
}

try
{
  m_listener.BeginAcceptTcpClient(DoAcceptTcpClientCallback, m_listener);
}
catch (Exception ex)
{
  s_log.Error(ex, "Exception BeginAccept");
}

Here it will authenticate the client/Server TLS handshake and get the SSL Stream. That will then be used to send and receive bytes. In the below code extract I am just writing the lines of code to obtain the SSL stream.

int timeout = 15000;
 // Create the SslStream using the client's network stream.
 var sslStream = new SslStream(client.GetStream(), false);

 // Authenticate the server but don't require the client to authenticate.
 Task serverAuth = sslStream.AuthenticateAsServerAsync(serverCertificate, false, SslProtocols.None, true);

 if (await Task.WhenAny(serverAuth, Task.Delay(timeout)) == serverAuth)
 {
   // task completed within timeout
   return sslStream;
 }
 else
 {
   // timeout logic
   throw new System.Exception("AuthenticateAsServerAsync timeout after " + timeout + " millisoconds");
 }

Problem Statement


In the normal scenario it is working fine. The Server stays up and running for weeks and months without any problem and millions of socket client connections are opened and then closed without any issue.

In a extremely rare situation AuthenticateAsServerAsync function does not obtain the SSL stream. So after 15 seconds I am timing out. But after this the listener ALSO stops accepting the new connections and even though the server is up and running no new clients can connect with the server. This is my original problem.

As this was occurring very rare so I have written a Test application in the .NET C# which is TCP/IP client making lots of connection requests to the server. Approximately 10 per seconds. I am stopping and starting this test application via a batch file. Using this test application (after running for about 3-4 hours) - I am able to terminate the Server process.

Here is what is happening

client = m_listener.EndAcceptTcpClient(ar);

The above line is producing the following exception

2021-04-20 19:48:19.4724|ERROR|OvationServerLib.RelayConnections.DeviceServer.DoAcceptTcpClientCallback: Exception EndAcceptTcpClient System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndAccept(Byte[]& buffer, Int32& bytesTransferred, IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.EndAccept(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpListener.EndAcceptTcpClient(IAsyncResult asyncResult)
   at OvationServerLib.RelayConnections.DeviceServer.DoAcceptTcpClientCallback(IAsyncResult ar) in C:\Development\Relay Server\fb4_relay_server\FB4_Relay_Server\Fb4RelayServer\OvationServerLib\RelayConnections\DeviceServer.cs:line 54

However, In the Event Viewer I am seeing slightly different exception, see below

Application: OvationServerService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
   at OvationServerLib.RelayConnections.DeviceServer.DoAcceptTcpClientCallback(System.IAsyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr)
   at System.Net.ContextAwareResult.CompleteCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Net.ContextAwareResult.Complete(IntPtr)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

So my questions are:

1- Why my server is terminating even though when EndAcceptTcpClient is producing an exception and I am already handling the exception.

2- Why timing out the AuthenticateAsServerAsync after 15 seconds is stopping the listener to accept new connections. I have to put some timeout as we cannot wait forever on this API.

c#
multithreading
sockets
ssl
tcpclient
asked on Stack Overflow Apr 21, 2021 by Zeshan Qureshi

1 Answer

1

Why my server is terminating even though when EndAcceptTcpClient is producing an exception and I am already handling the exception.

You are handling (and logging) the SocketException. But then your DoAcceptTcpClientCallback just continues running with client set to null. Instead of continuing, you should just return from the callback (since the socket is no longer viable).

Why timing out the AuthenticateAsServerAsync after 15 seconds is stopping the listener to accept new connections. I have to put some timeout as we cannot wait forever on this API.

Whenever you have a timeout on a socket operation, you should close the socket. This is especially true of "pessimistic" timeouts like Task.WhenAny. That kind of "timeout" doesn't actually stop the AuthenticateAsServerAsync call; you need to close the socket to force the AuthenticateAsServerAsync operation to stop.

answered on Stack Overflow Apr 21, 2021 by Stephen Cleary

User contributions licensed under CC BY-SA 3.0