TFS 2012 continuous integration deployment fails - Directory not empty

Sporadically get the error

An error was encountered when processing operation 'Delete Directory' on

'D:\xxx'. The error code was 0x80070091. The directory is not empty.

I'm using web deploy WMSVC and the user has full access to the root of the site (set up through IIS > Configure Web Deploy Publishing

EDIT
Or less frequently:

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets (4377): Web deployment task failed. (Unable to perform the operation ("Delete File") for the specified directory ("D:\xxx\bin\Microsoft.Practices.ServiceLocation.dll"). This can occur if the server administrator has not authorized this operation for the user credentials you are using. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_INSUFFICIENT_ACCESS_TO_SITE_FOLDER.)

I follow the instructions for the user (a domain user) but it makes no difference. The publish settings that get saved have publishMethod="MSDeploy" instead of WMSVC.

Note, it deploys correctly 95% of the time.

I know this sounds stupid, but do you have those folders open (or children thereof) on any computer? Especially if you had a dead remote desktop session or someone accessing it over the network share it will get a denial to be able to delete the directory.

Burned me a number of times, you're issue may be different though. I realize this is a late response, but found it while searching for my own issue and though I cuold help.