Daily bizarre network dropouts

1

We have a Hyper-V-hosted Windows Server 2008 R2 Standard server with Remote Desktop Services which has a long-standing problem whereby, every day in the early hours of the morning, its networking encounters lots of strange problems:

  1. Using Command Prompt to nslookup and ping FQDNs both public (www.google.co.uk) and private (%domainControllerHostname%.%activeDirectoryDNSDomainName%) succeed.
  2. Using Internet Explorer to browse to websites (http://support.me, https://www.google.co.uk, etc) fails with errors:

    • Pre-advanced reset: "Unable to open the search page".
    • Post-advanced reset: "This page can't be displayed".

  3. Using Explorer to browse to the domain controller server fails with error:

    "Network Error Windows cannot access \%domainControllerHostname%\"

  4. Using Explorer to browse to itself fails with error:

    "\127.0.0.1\ The name limit for the local computer network adapter card was exceeded".

  5. Using Command Prompt to execute command "nslookup -type=all _ldap._TCP.dc._msdcs.%activeDirectoryDNSDomainName%" succeeds.

  6. Error and warning events are logged:

Log Name: System
Source: Microsoft-Windows-GroupPolicy
Date: 14/07/2016 01:37:12
Event ID: 1055
Task Category: None
Level: Error
Keywords:
User: %activeDirectoryNetBIOSDomainName%\%activeDirectoryDomainAdministratorUsername%
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
The processing of Group Policy failed. Windows could not resolve the computer name. This could be caused by one of more of the following:
a) Name Resolution failure on the current domain controller.
b) Active Directory Replication Latency (an account created on another domain controller has not replicated to the current domain controller).

 

Log Name: System
Source: Microsoft-Windows-TerminalServices-RemoteConnectionManager
Date: 14/07/2016 03:02:19
Event ID: 1061
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
Remote Desktop Session Host server was unable to retrieve users Licensing information from AD. Error 0x8007054b.

 

Log Name: System
Source: NETLOGON
Date: 14/07/2016 03:32:12
Event ID: 5719
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
This computer was not able to set up a secure session with a domain controller in domain %activeDirectoryNetBIOSDomainName% due to the following:
The RPC server is unavailable.
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

ADDITIONAL INFO
If this computer is a domain controller for the specified domain, it sets up the secure session to the primary domain controller emulator in the specified domain. Otherwise, this computer sets up the secure session to any domain controller in the specified domain.

 

Log Name: System
Source: Microsoft-Windows-TerminalServices-Licensing
Date: 14/07/2016 03:35:38
Event ID: 4105
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
The Remote Desktop license server cannot update the license attributes for user "NOC_HelpDesk" in the Active Directory Domain "%activeDirectoryDNSDomainName%". Ensure that the computer account for the license server is a member of Terminal Server License Servers group in Active Directory domain "%activeDirectoryDNSDomainName%". If the license server is installed on a domain controller, the Network Service account also needs to be a member of the Terminal Server License Servers group.
If the license server is installed on a domain controller, after you have added the appropriate accounts to the Terminal Server License Servers group, you must restart the Remote Desktop Licensing service to track or report the usage of RDS Per User CALs.
Win32 error code: 0x8007203a

 

Log Name: System
Source: TermDD
Date: 14/07/2016 04:53:14
Event ID: 56
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
The Terminal Server security layer detected an error in the protocol stream and has disconnected the client. Client IP: 172.16.50.100.

 

Log Name: System
Source: TermDD
Date: 14/07/2016 04:53:24
Event ID: 50
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: %RDSServerHostname%.%activeDirectoryDNSDomainName%
Description:
The RDP protocol component X.224 detected an error in the protocol stream and has disconnected the client.

 

These problems can be temporarily resolved for ~24 hours by rebooting the server.

VMs hosted on the same Hyper-V host and even in the same network and Active Directory domain have no problems.

networking
windows-server-2008-r2
hyper-v
rds
asked on Server Fault Sep 23, 2016 by mythofechelon • edited Sep 26, 2016 by mythofechelon

1 Answer

0

After much diagnostics (including cloning the VM onto a different Hyper-V server, adding a 2012 R2 RDS server to the same AD domain, etc), our Continuum NOC determined the cause.

Bizarrely, the Windows service A300_Service (application TotalTimePlus version 7.00.0000) seems to be the cause - stopping it instantly resolves all problems without rebooting.

Here is the relevant section from their email containing further technical details:

I can see in the Process Explorer this service is always try to send the SYN packet to either of these IPs 192.168.39.218:5010 or 192.168.37.180:5010. Once the SYN packet is sent might be acknowledged at the destination. The process gets completed.

Below is the Process Explorer Logs TCP-IP SnapShot

NETSTAT Logs: I was not able to see this Application packet requests at the time of failures. These are the logs when I killed the Process and re-executed the service from the Service Console.

[A300_Service.exe] TCP 192.168.38.4:55275 192.168.37.180:5010 SYN_SENT
[A300_Service.exe] TCP 192.168.38.4:55276 192.168.39.218:5010 SYN_SENT
[A300_Service.exe] TCP 192.168.38.4:55277 192.168.0.218:5010 SYN_SENT
[A300_Service.exe] TCP 192.168.38.4:55278 192.168.37.180:5010 SYN_SENT
[A300_Service.exe] TCP 192.168.38.4:55279 192.168.39.218:5010 SYN_SENT

answered on Server Fault Sep 23, 2016 by mythofechelon • edited Sep 26, 2016 by mythofechelon

User contributions licensed under CC BY-SA 3.0