VSPerf and Azure Cloud Services

Same issue, 2 processes of VSPerf.exe burning 100% of CPU on 2nd role instance. IIS becomes unresponsive on that instance. We had the exact same issue 4 months back, disappeared "somehow" when profiling was enabled just when we were debugging this issue with MSFT support (someone inside MSFT knows this issue). But without being able to repro the issue (we only had screenshots) it was dropped.

Since I see it again, 5 minutes back and since Azure SDK 2.0 makes it easy to grab diagnostic logs (unlike the configuration ritual from SDK 1.8), here is something useful for the next folks to make sense off

The core error is Application VsPerf Tool Error: Error starting data collection with a dedicated process D:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Performance Tools\VSPerf. with the detailed logs (4 entries) below. Interestingly I had 2 processes of VSPerf.exe burning 100% of CPU and also had 2 log entries ....

{555908d1-a6d7-4695-8e1e-26931d2012f4}  Service Control Manager 7011    System  A timeout (30000 milliseconds) was reached while waiting for a transaction response from the iphlpsvc service.  <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Service Control Manager' Guid='{555908d1-a6d7-4695-8e1e-26931d2012f4}' EventSourceName='Service Control Manager'/><EventID Qualifiers='49152'>7011</EventID><Version>0</Version><Level>2</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8080000000000000</Keywords><TimeCreated SystemTime='2013-06-23T17:37:25.557218700Z'/><EventRecordID>1269</EventRecordID><Correlation/><Execution ProcessID='520' ThreadID='1104'/><Channel>System</Channel><Computer>RD00155DA909B8</Computer><Security/></System><EventData><Data Name='param1'>30000</Data><Data Name='param2'>iphlpsvc</Data></EventData></Event>   WebApp_IN_1 6/23/2013 5:37 PM   2   520 1104
{555908d1-a6d7-4695-8e1e-26931d2012f4}  Service Control Manager 7011    System  A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Schedule service.  <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Service Control Manager' Guid='{555908d1-a6d7-4695-8e1e-26931d2012f4}' EventSourceName='Service Control Manager'/><EventID Qualifiers='49152'>7011</EventID><Version>0</Version><Level>2</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8080000000000000</Keywords><TimeCreated SystemTime='2013-06-23T17:37:56.259560100Z'/><EventRecordID>1270</EventRecordID><Correlation/><Execution ProcessID='520' ThreadID='1104'/><Channel>System</Channel><Computer>RD00155DA909B8</Computer><Security/></System><EventData><Data Name='param1'>30000</Data><Data Name='param2'>Schedule</Data></EventData></Event>   WebApp_IN_1 6/23/2013 5:37 PM   2   520 1104
00000000-0000-0000-0000-000000000000    VSPERF  4100    Application VsPerf Tool Error: Error starting data collection with a dedicated process D:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Performance Tools\VSPerf. <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='VSPERF'/><EventID Qualifiers='57635'>4100</EventID><Level>2</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2013-06-23T17:40:37.000000000Z'/><EventRecordID>266</EventRecordID><Channel>Application</Channel><Computer>RD00xxxxxxxxx</Computer><Security/></System><EventData><Data>Error starting data collection with a dedicated process D:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Performance Tools\VSPerf.</Data></EventData></Event> WebApp_IN_1 6/23/2013 5:40 PM   2   0   0
00000000-0000-0000-0000-000000000000    VSPERF  4100    Application VsPerf Tool Error: Error starting data collection with a dedicated process D:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Performance Tools\VSPerf. <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='VSPERF'/><EventID Qualifiers='57635'>4100</EventID><Level>2</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2013-06-23T17:40:41.000000000Z'/><EventRecordID>267</EventRecordID><Channel>Application</Channel><Computer>RD00xxxxxxxxx</Computer><Security/></System><EventData><Data>Error starting data collection with a dedicated process D:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Performance Tools\VSPerf.</Data></EventData></Event> WebApp_IN_1 6/23/2013 5:40 PM   2   0   0

The closest I've found to a workaround is to create a startup task

<Task commandLine="VSPerf.cmd" executionContext="elevated" taskType="background" />

where VSPerf.cmd is a simple launcher

PowerShell -ExecutionPolicy Unrestricted .\VSPerf.ps1
exit /B 0

for a PowerShell script which attempts to kill over-active VSPerf instances:

function Restart-VSPerf {
    $procs = (
        gwmi -Query "SELECT IDProcess,PercentProcessorTime FROM Win32_PerfFormattedData_PerfProc_Process WHERE Name='vsperf' AND PercentProcessorTime > 25" |
        Foreach-Object { $_pid = $_.IDProcess ; gwmi -Query "SELECT Handle,CommandLine FROM win32_process WHERE Name='vsperf.exe' AND Handle=$_pid" | Select-Object Handle, CommandLine }
    )
    if ($procs) {
        # Get VSPerf path.
        # NB The regex uses conditional matching with an expression, as described in https://msdn.microsoft.com/en-us/library/36xybswe.aspx
        # to ensure that a single capturing group gets the contents of quoted arguments and unquoted arguments.
        $regex = [regex] '^(("?)((?(?<=")[^"]*|[^"\s]*))\2(\s+|$))+$'
        $vsperf = $regex.Match($procs[0].CommandLine).Groups[3].Captures[0].Value

        & $vsperf /detach

        # Kill
        $procs | Foreach-Object { Stop-Process -Force -Id $_.Handle }

        # Restart
        Start-Sleep -s 1
        $procs | Foreach-Object {
            $_args = ($regex.Match($_.CommandLine).Groups[3].Captures[1..99] |  Select-Object -ExpandProperty Value)
            Start-Process $vsperf -ArgumentList "$_args"
        }
    }
}

while (1) {
    Start-Sleep -s 180
    Restart-VSPerf
}

It works sometimes but not always. Suggestions for reliability improvements welcome.