Exchange 2010 Outlook Anywhere not working - RPC Proxy, Address Book?

Ugh. We built a greenfield AD and Exchange 2010 environment a few months back for a large client. Outlook Anywhere isn't working. Outlook clients just never connect, doesn't seem to be an error or timeout.

This was a migration. Originally we had a cert mismatch for Outlook Anywhere so I'm fairly certain it never worked. We recently changed everyone's SMTP domain suffix to match the webmail URL's domain, so I expected that OA would start working. Now we have a snow day, lots of people at home, and it turns out it doesn't work. Looks like there's some sort of problem with proxied RPC connecting to OAB but I'm quite stumped as to how to fix this.

Autodiscover is good, OWA, ActiveSync, EWS all have been working for months.

We have 2 mailbox servers in a DAG, 1 mailbox server for public folders and journals, a 2-server CAS array. On the inside, we have NLB for the array. On the outside, we have a simply Netscaler LB VIP for the array that does nothing special, just a TCP 443 passthrough.

RCA shows this - it's always about 33 seconds so I assume there's a 30-second timeout here.

Attempting to ping RPC proxy webmail.5ssl.com. RPC Proxy was pinged successfully. Additional Details Elapsed Time: 201 ms.

Attempting to ping the MAPI Mail Store endpoint with identity: PFEXCAS.internal.domain.name.com:6001. The attempt to ping the endpoint failed. Additional Details An RPC error was thrown by the RPC Runtime process. Error 1818 CallCancelled

Elapsed Time: 33536 ms.

Now, both CAS servers are indeed listening on TCP 6001. PFEXCAS is the name of the NLB VIP.

This works for ports 6001, 6002, 6004, so that's good

rpcping -t ncacn_http -s pfexcas.internal.domain.name.com -o RpcProxy=pfexcas.internal.domain.name.com -P " , , " -I " , , " -H 1 -u 10 -a connect -F 3 -v 3 -e 6004

This does not, it gives a 1722

rpcping -t ncacn_http -s pfexcas.internal.domain.name.com -o RpcProxy=webmail.5ssl.com -P " , , " -I " , , " -H 1 -u 10 -a connect -F 3 -v 3 -e 6004

Test-OutlookConnectivity -Protocol:Http -verbose works just fine on both servers, including the GetNewDSA and RFRI::GetReferral for the Address Book, those fail in our next test.

This is Ex2010, so HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\RpcProxy\ValidPorts_AutoConfig_Exchange has been checked and looks correct on both CAS servers

PFEXCAS:6001-6004;PFEXCAS.internal.domain.name:6001-6004;PFEXCAS01:6001-6004;PFEXCAS01.internal.domain.name:6001-6004;PFEXCAS02:6001-6004;PFEXCAS02.internal.domain.name:6001-6004;PFEXMB01:6001-6004;PFEXMB01.internal.domain.name:6001-6004;PFEXMB02:6001-6004;PFEXMB02.internal.domain.name:6001-6004;PFEXMB03:6001-6004;PFEXMB03.internal.domain.name:6001-6004

Running Test-OutlookConnectivity -RpcProxyTestType:Internal -RpcTestType:Server -verbose gives us this (results same/similar when run on either CAS server. So there's something wrong with RPC proxying to the address book but hell if I can figure out what. Everything else in the environment seems to be working just fine, and this setup was configured quite closely to the original environment which had working OA.

VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Target Client Access server = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Using connection parameters : 
'internal.domain.name.com\extest_23e99fa922cb4: RpcProxy/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic]'
VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Pinging RpcProxy at the following URL: 
https://PFEXCAS01.internal.domain.name.com/rpc/RpcProxy.dll.

ClientAccessServer   ServiceEndpoint                               Scenario                            Result  Latency
                                                                                                                  (MS)
------------------   ---------------                               --------                            ------  -------
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com                     RpcProxy::VerifyRpcProxy.           Success   46.80
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : Using connection parameters : 
'internal.domain.name.com\extest_23e99fa922cb4: Rfri/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic], 
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:13.714 GMT] Test-OutlookConnectivity : Mailbox = 'internal.domain.name.com\extest_23e99fa922cb4', Owner = 
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:13.714 GMT] Test-OutlookConnectivity : Address Book Operation = 'GetNewDSA', Input = 'userLegacyDN', 
Input Value = '/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba 
(The RPC server is unavailable) from cli_RfrGetNewDSA 

EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:21.342

EEInfo: Generating component: 2

EEInfo: Status: 0x000006BA

EEInfo: Detection location: 1710

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 1

EEInfo:    prm[0]: Long val: 0 (0x00000000)



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:21.342

EEInfo: Generating component: 13

EEInfo: Status: 0x000006BA

EEInfo: Detection location: 1352

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 1

EEInfo:    prm[0]: Long val: -1073606646 (0xC002100A)



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:21.342

EEInfo: Generating component: 14

EEInfo: Status: 0xC002100A

EEInfo: Detection location: 1380

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 12175 (0x00002F8F)

EEInfo:    prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6002



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:21.342

EEInfo: Generating component: 14

EEInfo: Status: 0x00010000

EEInfo: Detection location: 1385

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 16 (0x00000010)

EEInfo:    prm[1]: Binary Buffer: 
<junk>



   at ThrowRpcException(Int32 rpcStatus, String message)

   at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)

   at Microsoft.Exchange.Rpc.Rfri.RfriRpcClient.GetNewDSA(String userDN, String& server)

   at Microsoft.Exchange.Monitoring.RfriClientWrapper.<>c__DisplayClass7.<GetNewDSA>b__6()

   at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : AddressBook operation failed.  Operation = 'GetReferral', Error 
= 'The Address Book Referral operation failed. Operation = '<GetNewDSA>b__6', returnValue = '1722', server = 
'PFEXCAS01.internal.domain.name.com', user = '/o=FIVESTAR/ou=Exchange Administrative Group 
(FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb', .'
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com                     RFRI::GetReferral.                  Failure   -1.00
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Using connection parameters : 
'internal.domain.name.com\extest_23e99fa922cb4: Nspi/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic], 
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Address Book Operation = 'Bind'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba 
(The RPC server is unavailable) from cli_NspiBind 

EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:28.955

EEInfo: Generating component: 2

EEInfo: Status: 0x000006BA

EEInfo: Detection location: 1710

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 1

EEInfo:    prm[0]: Long val: 0 (0x00000000)



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:28.955

EEInfo: Generating component: 13

EEInfo: Status: 0x000006BA

EEInfo: Detection location: 1352

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 1

EEInfo:    prm[0]: Long val: -1073606646 (0xC002100A)



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:28.955

EEInfo: Generating component: 14

EEInfo: Status: 0xC002100A

EEInfo: Detection location: 1380

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 12175 (0x00002F8F)

EEInfo:    prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6004



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:28.955

EEInfo: Generating component: 14

EEInfo: Status: 0x00010000

EEInfo: Detection location: 1385

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 16 (0x00000010)

EEInfo:    prm[1]: Binary Buffer: 
<junk>



   at ThrowRpcException(Int32 rpcStatus, String message)

   at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)

   at Microsoft.Exchange.Rpc.Nspi.NspiRpcClient.Bind(UInt32 flags, IntPtr stat, IntPtr guid)

   at Microsoft.Exchange.Monitoring.NspiClientWrapper.<Bind>b__4()

   at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Address Book operation failed. Details: The NSPI operation 
failed. Operation = '<Bind>b__4', ReturnValue = '1722', Server = 'PFEXCAS01.internal.domain.name.com', User = 
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb', ..
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com                     NSPI::GetProfileDetails.            Failure   -1.00
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Using connection parameters : 
'internal.domain.name.com\extest_23e99fa922cb4: Mailbox/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic], 
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Mailbox Operation: 'EcDoConnectEx', Target mailbox: 
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:28.986 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.986 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba 
(The RPC server is unavailable) from EcDoConnectEx 

EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:36.599

EEInfo: Generating component: 13

EEInfo: Status: 0x000006BA

EEInfo: Detection location: 1352

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 1

EEInfo:    prm[0]: Long val: -1073606646 (0xC002100A)



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:36.599

EEInfo: Generating component: 14

EEInfo: Status: 0xC002100A

EEInfo: Detection location: 1380

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 12175 (0x00002F8F)

EEInfo:    prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6001



EEInfo: ComputerName: n/a

EEInfo: ProcessID: 4520

EEInfo: Generation Time: 2015-01-27 19:08:36.599

EEInfo: Generating component: 14

EEInfo: Status: 0x00010000

EEInfo: Detection location: 1385

EEInfo: Flags: 0

EEInfo: NumberOfParameters: 2

EEInfo:    prm[0]: Long val: 16 (0x00000010)

EEInfo:    prm[1]: Binary Buffer: 
<junk>



   at ThrowRpcException(Int32 rpcStatus, String message)

   at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)

   at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnect_Wrapped(Boolean isEx, Void** pcxh, Byte* 
szUserDN, UInt32 ulFlags, UInt32 ulConMod, UInt32 cbLimit, UInt32 ulCpid, UInt32 ulLcidString, UInt32 ulLcidSort, 
UInt32 ulIcxrLink, UInt16 usFCanConvertCodePages, UInt32* pcmsPollsMax, UInt32* pcRetry, UInt32* pcmsRetryDelay, 
UInt16* picxr, Byte** pszDNPrefix, Byte** pszDisplayName, UInt16* rgwClientVersion, UInt16* rgwServerVersion, UInt16* 
rgwBestVersion, UInt32* pulTimeStamp, Byte* rgbAuxIn, UInt32 cbAuxIn, Byte* rgbAuxOut, UInt32* pcbAuxOut)

   at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnect_Internal(Boolean isEx, IntPtr& contextHandle,
 String userDn, Int32 flags, Int32 sizeLimit, Int32 conMod, Int32 cpid, Int32 lcidString, Int32 lcidSort, Int32 
sessionIdLink, Boolean canConvertCodePages, TimeSpan& pollsMax, Int32& retryCount, TimeSpan& retryDelay, String& 
dnPrefix, String& displayName, Int16[] clientVersion, Int16[]& serverVersion, Int16[]& bestVersion, Int32& timeStamp, 
Byte[] auxIn, Boolean compressAuxIn, Boolean xorMagicAuxIn, Byte[]& auxOut, Int32 maxSizeAuxOut, Boolean& 
wasCompressedAuxOut, Boolean& wasXorMagicAuxOut)

   at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnectEx(IntPtr& contextHandle, String userDn, Int32
 flags, Int32 sizeLimit, Int32 conMod, Int32 cpid, Int32 lcidString, Int32 lcidSort, Int32 sessionIdLink, Boolean 
canConvertCodePages, TimeSpan& pollsMax, Int32& retryCount, TimeSpan& retryDelay, String& dnPrefix, String& 
displayName, Int16[] clientVersion, Int16[]& serverVersion, Int16[]& bestVersion, Int32& timeStamp, Byte[] auxIn, 
Boolean compressAuxIn, Boolean xorMagicAuxIn, Byte[]& auxOut, Int32 maxSizeAuxOut, Boolean& wasCompressedAuxOut, 
Boolean& wasXorMagicAuxOut)

   at Microsoft.Exchange.Monitoring.MailboxTask.<>c__DisplayClassb.<Connect>b__a()

   at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : EcDoConnectEx Result: '1722'
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com                     Mailbox::Connect.                   Failure   -1.00
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : This step was skipped. This may have been caused by the failure 
in a previous scenario. Operation = 'MailboxTask::Logon'.
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com                     Mailbox::Logon.                     Skipped   -1.00
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : Admin Audit Log: Entered Handler:OnComplete.
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : Ending processing &

This was a stupid configuration Heisenbug of some kind. The registry was configured properly for RPC, and the problem was not specific to the OAB, that's just the first thing that's checked. It took a reboot of the CAS servers to get it to work.

Further, the real root cause was that I had recently installed a third CAS server as a test, not in the array. I fed it dummy values for public URLs that weren't reachable, and for some reason (wish I knew) those got written into Autodiscover. I uninstalled the CAS role from that test machine and it cleaned up autodiscover, but still had trouble with RPCPing until the reboot of the remaining CAS servers.