Msdeploy fails from time to time, stating that it is unable to delete directory

We have an MsDeploy enabled-delivery and, normally, it is working flawlessly on number of servers. But, on 3 Windows 2012 R2 servers it fails from time to time saying that it is enable to delete directory. Following error is logged in Event log:

User: 
Client IP: masked

Content-Type: application/msdeploy
Version: 9.0.0.0
MSDeploy.VersionMin: 7.1.600.0
MSDeploy.VersionMax: 9.0.1764.0
MSDeploy.Method: Sync
MSDeploy.RequestId: 41e11d1d-ab5d-457e-9daf-7bff7655d6ac
MSDeploy.RequestCulture: en-US
MSDeploy.RequestUICulture: en-US
ServerVersion: 9.0.1763.0
Skip: objectName="^configProtectedData$"objectname="filePath",absolutepath="app_offline-template.htm"objectname="filePath",absolutepath="app_offline.htm"
Provider: auto, Path: 
A tracing deployment agent exception occurred that was propagated to the client. Request ID '41e11d1d-ab5d-457e-9daf-7bff7655d6ac'. Request Timestamp: '10/20/2015 9:21:42 AM'. Error Details:
Microsoft.Web.Deployment.DeploymentClientServerException: An error was encountered when processing operation 'Delete Directory' on 'some_path'. ---> Microsoft.Web.Deployment.DeploymentException: The error code was 0x80070091. ---> System.IO.IOException: The directory is not empty.

   at Microsoft.Web.Deployment.NativeMethods.RaiseIOExceptionFromErrorCode(Win32ErrorCode errorCode, String maybeFullPath)
   at Microsoft.Web.Deployment.DirectoryEx.Delete(String path)
   at Microsoft.Web.Deployment.DirPathProviderBase.Delete(Boolean whatIf)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Web.Deployment.FilePathProviderBase.HandleKnownRetryableExceptions(DeploymentBaseContext baseContext, Int32[] errorsToIgnore, Exception e, String path, String operation)
   at Microsoft.Web.Deployment.DirPathProviderBase.Delete(Boolean whatIf)
   at Microsoft.Web.Deployment.DeploymentObject.DeleteOperation(DeploymentSyncContext syncContext)
   at Microsoft.Web.Deployment.DeploymentObject.Delete(DeploymentSyncContext syncContext)
   at Microsoft.Web.Deployment.DeploymentSyncContext.HandleDelete(DeploymentObject destObject, DeploymentObject sourceParentObject)
   at Microsoft.Web.Deployment.DeploymentSyncContext.SyncDirPathChildren(DeploymentObject destRoot, DeploymentObject sourceRoot)
   at Microsoft.Web.Deployment.DeploymentSyncContext.SyncDirPathChildren(DeploymentObject destRoot, DeploymentObject sourceRoot)
   at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
   at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
   at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
   at Microsoft.Web.Deployment.DeploymentSyncContext.ProcessSync(DeploymentObject destinationObject, DeploymentObject sourceObject)
   at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable`1 syncPassId, String syncSessionId)
   at Microsoft.Web.Deployment.DeploymentAgent.HandleSync(DeploymentAgentAsyncData asyncData, Nullable`1 passId)
   at Microsoft.Web.Deployment.DeploymentAgent.HandleRequestWorker(DeploymentAgentAsyncData asyncData)
   at Microsoft.Web.Deployment.DeploymentAgent.HandleRequest(DeploymentAgentAsyncData asyncData)

All deployments start from putting app_offline.htm to root of web app, and it is deleted as separate step - so, application will not be able to start. Path is different each time. If deployment is restarted immediately - it will fail (despite of application being offline due to App_offline.htm) The only way (found so far) to reach successful deployment - restart Management Service through IIS.

Sometimes it helps to restart management service before starting deployment.

I fixed it by setting SkipExtraFilesOnServer to True - thus, I have not clean file system on server, but builds are running ok