Forefront Identity Manager errors when starting the SharePoint 2010 User Profile Synchronization Service Application

After following Spence Harbar's Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization (http://www.harbar.net/articles/sp2010ups.aspx) exactly (and having it work several times for other implementations) in this particular instance, starting the user profile synchronization service generates the following errors (these can be found in the windows logs of the server attempting to start and run the UPS Sync) and eventually fails:

Error ID: 22 - The Forefront Identity Manager Service cannot connect to the SQL Database Server.

The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.

Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.

AND

Error ID: 3 - .Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered

at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException) at Microsoft.ResourceManagement.Data.DataAccess.RetrieveWorkflowDataForHostActivator(Int16 hostId, Int16 pingIntervalSecs, Int32 activeHostedWorkflowDefinitionsSequenceNumber, Int16 workflowControlMessagesMaxPerMinute, Int16 requestRecoveryMaxPerMinute, Int16 requestCleanupMaxPerMinute, Boolean runRequestRecoveryScan, Boolean& doPolicyApplicationDispatch, ReadOnlyCollection1& activeHostedWorkflowDefinitions, ReadOnlyCollection1& workflowControlMessages, List`1& requestsToRedispatch) at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RetrieveWorkflowDataForHostActivator() at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHosts(Object source, ElapsedEventArgs e)

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlDataReader.ConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader() at Microsoft.ResourceManagement.Data.DataAccess.RetrieveWorkflowDataForHostActivator(Int16 hostId, Int16 pingIntervalSecs, Int32 activeHostedWorkflowDefinitionsSequenceNumber, Int16 workflowControlMessagesMaxPerMinute, Int16 requestRecoveryMaxPerMinute, Int16 requestCleanupMaxPerMinute, Boolean runRequestRecoveryScan, Boolean& doPolicyApplicationDispatch, ReadOnlyCollection1& activeHostedWorkflowDefinitions, ReadOnlyCollection1& workflowControlMessages, List`1& requestsToRedispatch)

AND

Error ID: 234 - ILM Certificate could not be created.

(Note: There are several of these, one for each stage of ILM certificate creation failure).

The system is a multi (3) server farm:

WFE Windows 2008 64Bit SharePoint 2010

APP (UPS and UPS Sync Running Here) Windows 2008 64Bit SharePoint 2010

SQL (Default Instance) SQL Server 2008 R2 64 Bit

All the necessary steps in Spence's guide (domain accounts, permissions, rights, etc.) have been followed.

The two FIM services on the APP server are starting and are using the FARM account.

The ILMMA and MOSS- folders are NOT present in %Programfiles%\Microsoft Office Servers \14.0\Synchronization Service\MaData.

A similiar thread by others (no defined resolution) can be found here:

http://social.technet.microsoft.com/Forums/en-US/sharepoint2010setup/thread/bac36f2b-0d7b-4e88-830b-ebb0a85f111e

Thoughts? Suggestions? Solutions?

UPDATE

It turns out the Secure Store Service Application was deployed, but not configured and missing a key. Completing this took care of all of the errors above with the exception of the first:

Error 22: The Forefront Identity Manager Service cannot connect to the SQL Database Server.

This error now only exists once in the logs. The User Profile Sync service is now stuck at starting rather than fails. The FIM sync service doesnt start due to login failure (hence the error).

Restarting does nothing. As mentioned the steps in Spence's guide have been followed and the appropriate account is in the right groups with the right permissions.

Thoughts?

UPDATE

The SQL Server now has the following errors, repeatedly:

Event 17806

SSPI handshake failed with error code 0x8009030c, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure.

Event 18452

Login failed. The login is from an untrusted domain and cannot be used with Windows authentication.

UPDATE

Switching to mixed mode authentication (?) on SQL fixed these problems.

I then ensured a bunch of my other service applications were started and configured.

I then did a couple of reboots.

I then used powershell to unprovision the synchronization service.

I then started (provisioned) the synchronization service.

The original errors are back, though BOTH the FIM services are started on the APP server.

UPDATE

From the ULS, this seems to be the error that is my plague.

07/09/2010 13:09:02.06 OWSTIMER.EXE (0x04C4) 0x1398 SharePoint Portal Server User Profiles 9q15 High UserProfileApplication.SynchronizeMIIS: Failed to configure ILM, will attempt during next rerun. Exception: System.Runtime.InteropServices.COMException (0x80070035): The network path was not found. at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail) at System.DirectoryServices.DirectoryEntry.Bind() at System.DirectoryServices.DirectoryEntry.get_AdsObject() at System.DirectoryServices.PropertyValueCollection.PopulateList() at System.DirectoryServices.PropertyValueCollection..ctor(DirectoryEntry entry, String propertyName) at System.DirectoryServices.PropertyCollection.get_Item(String propertyName) at Microsoft.Office.Server.Administration.UserProfileApplication.AddAccountToMIISUsersList(String strAccount, Hashtable htPermittedUsers, Hashtable htNewlyAddedUsers) at Microsoft.Office.Server.Administration.UserProfileApplication.SetupProfileSynchronizationEnginePermissions() at Microsoft.Office.Server.Administration.UserProfileApplication.SetupSynchronizationService(ProfileSynchronizationServiceInstance profileSyncInstance). 6d80e09e-5883-43c1-9ca0-2377646b6f00

I should add, WINS is enabled on the DCs and clients. So is local DTC access. I've even gone as far as to create a HOSTS file on the machines pointing to the right servers.

UPDATE

Everything is working. On a whim, I added my FARM account to the domain admin group for the provisioning process. Then I rebooted. Everything has started, the sync service in Central Admin AND the two FIM services.

Now, that SHOULDN'T have been required. But, it worked.

Anyway, things are working fine now.

If you end up doing this, don't forget to REMOVE your FARM account from the domain admin and local admin groups you have added it to.

Additionally, if you have this much trouble, it is recommended that you wipe everything and start fresh. A lot happens with the UPS service during provisioning and its better to have a clean system than a dirty functioning one.

SOLUTION (END)

These errors are related to UPS provisioning incorrectly. This is likely related to issues with the existing active directory in the environment, e.g. not being able to read appropriate AD objects, the network path not found is actually related to this. The UPS is a beast. If you run into these errors and have exhausted EVERY recommended avenue, assess your active directory. Check its functional level, policies, standard permissions etc. For example, we found in this implementation that initially, some user accounts did not have basic read permissions. Then, start over, cleanly.

UPDATE

Everything is working. On a whim, I added my FARM account to the domain admin group for the provisioning process. Then I rebooted. Everything has started, the sync service in Central Admin AND the two FIM services.

Now, that SHOULDN'T have been required. But, it worked.

Anyway, things are working fine now.

If you end up doing this, don't forget to REMOVE your FARM account from the domain admin and local admin groups you have added it to.

Additionally, if you have this much trouble, it is recommended that you wipe everything and start fresh. A lot happens with the UPS service during provisioning and its better to have a clean system than a dirty functioning one.

SOLUTION (END)

These errors are related to UPS provisioning incorrectly. This is likely related to issues with the existing active directory in the environment, e.g. not being able to read appropriate AD objects, the network path not found is actually related to this. The UPS is a beast. If you run into these errors and have exhausted EVERY recommended avenue, assess your active directory. Check its functional level, policies, standard permissions etc. For example, we found in this implementation that initially, some user accounts did not have basic read permissions. Then, start over, cleanly.