Why is appcmd.exe unlock config not working on Azure emulator?

I've recently upgraded to the Azure 2.1 SDK, and I'm now encountering a problem with part of my web.config in a web role when running on the compute emulator. My web.config contains this:

<location path="api">
  <system.webServer>
    <security>
      <access sslFlags="Ssl, SslRequireCert, SslNegotiateCert" />
    </security>
  </system.webServer>
  <system.web>
    <authorization>
      <allow users="*" />
    </authorization>
  </system.web>
</location>

I need this because everything under the /api/ path requires clients to authenticate by providing client-side certificates over HTTPS.

By default, IIS is configured not to allow you to do this - the <access> element under system.webServer/security is locked by default. So I've always had a startup task that contains this:

SET APPCMD=%windir%\system32\inetsrv\appcmd.exe
IF EXIST APPCMD GOTO :INUSUALPLACE
SET APPCMD="%ProgramFiles%\IIS Express\appcmd.exe"
:INUSUALPLACE
%APPCMD% unlock config /section:system.webServer/security/access

Without that, you would get a 500.19 error. Up until recently, this startup task has always successfully prevented that error, enabling my SSL configuration to work.

But it's no longer working, and as far as I can tell, this happened when I switched to the 2.1 SDK. Everything else in this web role works by the way - it's only when I try to access the services under the /api/ path to which the SSL configuration settings apply that I get an error. And it's a 500.19. (The 500 is 'internal server error' of course, but the .19 signifies that it's a configuration error.)

As far as I can tell, it's happening because the attempt to unlock this configuration section is no longer working. The reason I say that is that if I find the applicationHost.config file that the Azure emulator creates (in C:\Users\<user>\AppData\Local\dftmp\Resources\<some random guid>\temp\temp\RoleTemp) and I manually edit it, replacing the Deny for the security element with an Allow, I stop getting the error, and can successfully use the services that require client certificates.

That's no use as a workaround of course - this applicationHost.config is regenerated each time you run the app in the emulator (and the exact location changes each time). I need some way to reliably unlock this configuration section automatically each time I debug the application locally. That's what appcmd.exe is supposed to do, but it seems to have stopped working.

It did occur to me that the problem might be that it's picking up the IIS version of appcmd.exe, even though the Azure SDK now uses IIS express. I'm not sure if they're different programs, so I tried adding this at the end of my startup command:

"%ProgramFiles%\IIS Express\appcmd.exe" unlock config /section:system.webServer/security/access

This explicitly runs the IIS Express copy. But it doesn't seem to make any difference.

The startup task is definitely running, before anyone asks. In the same folder as the applicationHost.config, I see a WaHostBootstrapper.log file, and it contains (amongst other things) these lines:

[00025156:00018324, 2013/08/30, 22:15:03.033, INFO ] Executing Startup Task type=0 rolemodule=(null) cmd="c:\dev\mm\DevInt\src\Mm.Cloud\csx\Debug\roles\Mm.Web\approot\bin\Startup\EnableClientCerts.cmd" 
[00025156:00018324, 2013/08/30, 22:15:03.034, INFO ] Executing "c:\dev\mm\DevInt\src\Mm.Cloud\csx\Debug\roles\Mm.Web\approot\bin\Startup\EnableClientCerts.cmd" .
[00025156:00018324, 2013/08/30, 22:15:03.221, INFO ] Program "c:\dev\mm\DevInt\src\Mm.Cloud\csx\Debug\roles\Mm.Web\approot\bin\Startup\EnableClientCerts.cmd"  exited with 0. Working Directory = c:\dev\mm\DevInt\src\Mm.Cloud\csx\Debug\roles\Mm.Web\approot\bin

This indicates that my EnableClientCerts.cmd (the script that calls appcmd.exe) ran without error.

I'm not entirely clear on how appcmd.exe knows which particular web site it's supposed to be configuring. There are several - I've got IIS proper on this box, and there's also a non-Azure-related IIS Express site configured. Is it possible that it's failing to configure the correct target?

Also, I see a few of this sort of error in the WaHostBootstrapper.log:

[00025156:00018324, 2013/08/30, 22:15:03.033, ERROR] <- WapGetEnvironmentVariable=0x800700cb

Could that be related?

Is there something missing from my script for unlocking the configuration section?

It turns out that when the emulator launches the startup task, it has already set the APPCMD variable. Moreover, it sets it not just to refer to the AppCmd.exe, it also includes a command line switch that points to the right configuration file:

"C:\Program Files\IIS Express\appcmd.exe" /apphostconfig:"C:\Users\Ian\AppData\Local\dftmp\Resources\1217ef49-a59a-4e18-8ebc-27d06a78cbd5\temp\temp\RoleTemp\applicationHost.config"

So if the startup script just uses %APPCMD% without first attempting to set it, it will apply to the correct instance. My script was not working because it made its own decision about where AppCmd.exe was located, and would end up modifying global settings for either IIS or IIS Express, neither of which seem to have any impact on the instance of IIS that the Azure Emulator hosts. (I'm guessing that this is a recent change in behaviour, possibly relating to the new feature in Azure SDK 2.1 that enables non-elevated development.)

The thing that worries me about this is that I can't find any documentation that mentions this pre-defined APPCMD variable. I only discovered it by adding the following to my startup command script:

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe "gci env: | format-list"  > c:\temp\env.log

I added that temporarily and ran the web role, and it provided a complete dump of all environment variables. Looking through that list, the APPCMD variable is the only one that contains the information required to target the correct configuration. But the documentation for startup tasks seems to recommend pointing directly at the IIS copy of AppCmd.exe - the Use AppCmd.exe to Configure IIS at Startup article just hardcodes the path. I guess that would work if I enabled the use of full IIS in the emulator, but I don't really want to do that.

So although this solution works (and appears to be the only viable solution, given what's in the startup task's environment) it makes me nervous because it's an undocumented feature. So be careful if you've stumbled upon this answer - it might not be reliable.