Web Management Service failing at random

so we have a custom deploy script that looks similar to this

    msdeploy.exe ^
      -source:manifest='path_to_manifest' ^
      -dest:auto,ComputerName="https://host:port/msdeploy.axd?site=sitename",UserName='domain\user',Password='muhpass',IncludeAcls='False',AuthType='Basic' ^
      -verb:sync ^
      -enableRule:DoNotDeleteRule ^
      -disableLink:AppPoolExtension ^
      -disableLink:ContentExtension ^
      -disableLink:CertificateExtension ^
      -setParamFile:"path_to_the_file" ^
      -allowUntrusted ^
      -useChecksum ^
      -retryAttempts=2 ^
      -preSync:runCommand="path_to_script",waitInterval=60000 ^
      -postSync:runCommand="path_to_script",waitInterval=60000

I have removed some of the values, but you can probably assume they're correct, because the scripts deploys the app successfully, just not every time. This is the step that fails:

    Performing '-preSync'...
    Info: Using ID '7bb88177-4329-4b66-a7b9-a563811edca3' for connections to the remote server.
    Error: (7/17/2013 2:27:00 PM) An error occurred when the request was processed on the remote computer.
    Error: An error occurred while communicating with the remote host. The error code is 0x80070006.
    Error: The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
    Error count: 1.
    Error during '-preSync'.

It usually completes withing seconds, but every once in a while it will take a really long time and eventually I get the error above. At this point I have to restart the Web Management Service on the production server, which usually fixes it till the next deploy. However, restarting the service also fails, but after it does, you can at least start the service manually and that's it.

If I go to the event viewer on the production server it has the following error under Microsoft Web Deploy (the time of the error as well as the id of the request match):

    User: 
    Client IP: muh_ip
    Content-Type: application/msdeploy
    Version: 9.0.0.0
    MSDeploy.VersionMin: 7.1.600.0
    MSDeploy.VersionMax: 9.0.1631.0
    MSDeploy.Method: Sync
    MSDeploy.RequestId: 7bb88177-4329-4b66-a7b9-a563811edca3
    MSDeploy.RequestCulture: en-US
    MSDeploy.RequestUICulture: en-US
    ServerVersion: 9.0.1631.0
    Skip: objectName="^configProtectedData$"
    Provider: auto, Path: 
    A tracing deployment agent exception occurred that was propagated to the client. Request ID '7bb88177-4329-4b66-a7b9-a563811edca3'. Request Timestamp: '7/17/2013 11:27:00 AM'. Error Details:
    System.Web.HttpException: An error occurred while communicating with the remote host. The error code is 0x80070006. ---> System.Runtime.InteropServices.COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
       --- End of inner exception stack trace ---
       at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
       at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
       at System.Web.HttpResponse.Flush(Boolean finalFlush)
       at System.IO.BufferedStream.FlushWrite()
       at System.IO.BufferedStream.Flush()
       at System.IO.BufferedStream.Dispose(Boolean disposing)
       at System.IO.Stream.Close()
       at System.IO.StreamWriter.Dispose(Boolean disposing)
       at System.IO.StreamWriter.Close()
       at System.Xml.XmlTextWriter.Close()
       at Microsoft.Web.Deployment.TraceEventStreamSerializer.Dispose(Stream stream)
       at Microsoft.Web.Deployment.DeploymentAgent.HandleSync(DeploymentAgentAsyncData asyncData, Nullable`1 passId)
       at Microsoft.Web.Deployment.DeploymentAgent.HandleRequestWorker(DeploymentAgentAsyncData asyncData)
       at Microsoft.Web.Deployment.DeploymentAgent.HandleRequest(DeploymentAgentAsyncData asyncData)

To me it definitely looks like there is some problem with this service, but I'm out of ideas, so I will appreciate any suggestions that may help solve this or at least gather more information. This thing is driving me nuts. It's also getting more frequent lately - used to happen less than once a week, now it's every time I deploy. I realize this post is getting really long, but I wanted to include all the relevant information.

Server is windows server 2008 R2 enterprise v6.1 sp1 with IIS 7.5.

tl;dr deploy script fails at random, restarting web management service fixes it temporary, wut do?