Socket servers on Windows VM on Azure stop listening on incomming requests

I have a few 100 installations of windows services running on the same Windows VM. Each of them are listening 2 ports each, for example 10050 and 60050.

Several times per day are one (or several) services no longer accept new incomming connections.

When I restart the service I get the following exception.

System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at System.Net.Sockets.TcpListener.Start(Int32 backlog)
   at SocketServerAsync.AsynchronousSocketListener.Start()

When I run netstat -ano I can't see anything using my port.

To get the service started again I must change the portnumber.

Edit 1 Each service listens to 2 port, one of them are a WCF service and the other is a TcpListener. When I restart them normally it works just fine. Most of the time its the TcpListener port that get used by something else but it does happen that the WCF service port is taken. Even if I have the service stopped for an hour the service are unable to start again once the problem occurs.

1. What is "taking" my ports?
2. Do I need to create endpoints for the VM?
3. Is there any other way to see what is blocking the port?
4. Can it be Azure itself that takes my ports randomly even if I'm already listening to them?

According to the error message, the reason for this problem is that you have 2 processes trying to use the port and get this problem.

When we use .NET to use resources of operation system, like file IO, network etc. These are all unmanaged resources, we need to release these resources correctly after we finish using them.

So I think the socket isn't getting shutdown properly and when the service restarts, it won't be able to bind to the port because windows may not know the process is done with it.

We should let our code to be more perfect and consider all abnormal situations.

We can shutdown socket properly using method TcpListener.Stop