Importing a DSA key from xml string fails for one user. Permissions? Broken installation? Bad KSP?

A user recently reported a weird error when using my software. I use DSA signatures to verify licenses. When the software imports the public key to verify a signature, the DSA provider's FromXmlString method throws a CryptographicException with the description "Key not valid for use in specified state."

It would appear that the _OpenCSP method called from System.Security.Cryptography.Utils.CreateProvHandle returns a NTE_BAD_KEY_STATE (0x8009000b). This is the first time anyone has reported this error to me, and that code has not changed for years.

What are the likely causes of this? A masked permissions error? A broken CAPI installation? Blocked by .net trust/permissions settings? Junk stored by a key storage provider, or a KSP returning something unexpected to cryptoapi?

I have googled the error code/description/etc but didn't come across any real answers as to what might cause this...

An isolated version of the code that fails is here: http://forum.huagati.com/getattachment.ashx?fileid=78

using System;
using System.Security.Cryptography;
using System.Reflection;

public class Test
{
  public static void Main()
  {
    try
    {
      string key = "<DSAKeyValue><P>wrjxUnfKvH/1s5cbZ48vuhTjflRT5PjOFnr9GeUPZSIoZhYATYtME4JRKrXBtSkyioRNtE1xgghbGAyvAJ5jOWw88fLBF+P1ilsZyq72G1YcbB+co8ImQhAbWKmdCicO9/66Th2MB+7kms/oY3NaCzKEuR7J3b23dGrFpp4ccMM=</P><Q>xmxoSErIJCth91A3dSMjC6yQCu8=</Q><G>bwOLeEaoJHwSiC3i3qk9symlG/9kfzcgrkhRSWHqWhyPAfzqdV1KxJboMpeRoMoFr2+RqqKHgcdbzOypmTeN4QI/qh4nSsl5iEfVerarBOrFuRdOVcJO0d8WE233XQznd1K66nXa5L8d9SNZrM6umZ1YuBjhVsTFdPlIXKfGYhk=</G><Y>wZnEEdMUsF3U3NBQ8ebWHPOp37QRfiBn+7h5runN3YDee1e9bC7JbJf+Uq0eQmU8zDs+avEgD68NpxTKEHGr4nQ3rW6qqacj5SDbwO7nI6eN3wWrVhvrWcQm0tUO93m64HsEJREohfoL+LjqgrqIjZVT4D1KXE+k/iAb6WKAsIA=</Y><J>+zmcCCNm2kn1EXH9T45UcownEe7JH+gl3Lw2lhVzXuX/dYp5sGCA2lK119iQ+m3ogjOuwABATCVFLo6J66DsSlMd0I8WSD5WKPvypQ7QjY0Iv71J2N0FW0ZXpMlk/CE8zq4Z7arM1N564mNe</J><Seed>QDrZrUFowquY5Uay8YtUFOXnv28=</Seed><PgenCounter>Gg==</PgenCounter></DSAKeyValue>";

      DSACryptoServiceProvider csp2 = new DSACryptoServiceProvider();
      csp2.FromXmlString(key);

      Console.WriteLine("Success!");
    }
    catch (Exception ex)
    {
      int hResult = 0;
      try
      {
          PropertyInfo pi = typeof(Exception).GetProperty("HResult", BindingFlags.NonPublic | BindingFlags.Instance);
          hResult = (int)pi.GetValue(ex, null);
      }
      catch (Exception ex2)
      {
          Console.WriteLine("HResult lookup failed: " + ex2.ToString());
      }
      Console.WriteLine("Initializing CSP failed: " + ex.ToString() + "\r\nHResult: " + hResult.ToString("x"));
    }
    Console.WriteLine("\r\nPress Enter to continue");
    Console.ReadLine();
  }
}

...and on the affected user's machine it returns:

Initializing CSP failed: System.Security.Cryptography.CryptographicException: Ke
y not valid for use in specified state.

at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters paramete
rs, Boolean randomKeyContainer)
at System.Security.Cryptography.Utils.get_StaticDssProvHandle()
at System.Security.Cryptography.DSACryptoServiceProvider.ImportParameters(DSA
Parameters parameters)
at System.Security.Cryptography.DSA.FromXmlString(String xmlString)
at Test.Main()
HResult: 8009000b

Update: The same code works fine when running under .net fx 2.0 on the same machine, but fails under .net fx 4.0.

Update 2: It appears that the DSA provider looks for keys stored under %APPDATA%\Microsoft\Crypto\DSS\[SID] even then initializing with an existing key. Could there be a conflict with this mechanism? Anyone know more about how that key storage thing operates, and why it is hit when loading a public key from a string?

The problem, which you describe, seems me very interesting, but without some additional information and some experiments it is difficult definitively to say what is the reason. So I try to describe how I understand the problem.

First of all .NET cryptography classes use internally the unmanaged CryptoAPI. So the method _OpenCSP call internally CryptAcquireContext function. In its the documentation we can read following about the error NTE_BAD_KEY_STATE (0x8009000BL):

The user password has changed since the private keys were encrypted.

Users private keys used by DSA provider are saved as files in the directory %APPDATA%\Microsoft\Crypto\DSS\[SID] and will be encrypted with relatively sophisticated algorithm about which you can read here. Important to understand that the files from the directory corresponds to the key containers of user's keys. Typically the user has full access to the files in the file system. The files will be encrypted with the key which depend on the user's password. In many standard cases the files will be re-encrypted after the password change, but the recovery algorithm, which depend on many things. If the password was reset instead of the changing by the user itself (by from an domain administrator/account operator and so on), the old contain of the directory %APPDATA%\Microsoft\Crypto\DSS\[SID] could be not more useful. For example if the user is not an Active Directory user (a local user) and the local administrator reset his password then the problem with crypto-containers will be take place.

So the first suggestion would be to ask the user whether his Active Directory password was reset. Next you should verify that the directory %APPDATA%\Microsoft\Crypto\DSS\[SID] exist in the user's profile and the user has full access to the directory in the file system. The you should delete all files from the directory (creating previously the backup copy of the files). By the way it is interesting to know whether the user has central saved profile (saved on the server). If it has central profile one can verify that the same problem, which you describe, exists on the other computer for the user and other user will has no problems on his original computer.

One more question which is not quite clear for me is why the key container from the the directory %APPDATA%\Microsoft\Crypto\DSS\[SID] are used at all because you use only public keys. In CryptoAPI one should use CryptAcquireContext with NULL as pszContainer parameter and CRYPT_VERIFYCONTEXT in dwFlags. I am not sure that .NET use the CRYPT_VERIFYCONTEXT flag and it could be indirect your problem.

You can create DSACryptoServiceProvider with the constructor having CspParameters parameter. CspParameters on the other side has Flags property which is extended in .NET 4.0 with the value CreateEphemeralKey. The description of CspProviderFlags.CreateEphemeralKey is very close to the description of the CRYPT_VERIFYCONTEXT flag of the CryptAcquireContext function. So use can try to use CspProviderFlags.CreateEphemeralKey or CspProviderFlags.CreateEphemeralKey together with CspProviderFlags.UseDefaultKeyContainer (NULL as pszContainer parameter of CryptAcquireContext means also default key container).

Moreover, if it is possible, you can try to debug the problem on the computer where the problem can be reproduced. For debugging you can use .NET sources which can be enabled (see here and here) or downloaded from here. You can then answer on some questions about the values of CspParameters which currently used in your program and compare the values for .NET 3.5 and .NET 4.0.

If what I wrote will not help to solve the problem, please could you append your question with additional informations:

• Which operation system and which service pack has the computer where the problem can be reproduced?
• Is the problem user dependent? I mean: has other users on the same computer the same problem?
• Is the the problem exist with domain (active directory) user or with the local user account? Has the user which has the problem central user profile which are saved on the server? If he has, than can the problem can be reproduced also on the other computers by the user?
• Could you describe a little more the environment in which situation you use the public key verification? Especially is the program runs in the user security context or you make some impersonation?

UPDATED: After reading of the forum where the problem originally was posted, I become pessimistic about solving of the problem. If you have no direct contact to the computer where the problem could be reproduced and the communication with the only user who has the problem are done only per posting in the forum... Nevertheless I have been thinking about the problem and so I decided to try to reproduce the problem myself. And I had success in this. So I'll describe here my results carefully. I'll describe how the can reproduce the problem so that it looks exactly like it is described in the forum thread.

You should do the following steps:

1) You create a local test account on your computer. 2) You login with the test account and generate the default key container for the DSA provider. You can do this for example with respect of the small .NET program which call following simple function:

static string GenerateDsaKeyInDefaultContainer()
{
    const int PROV_DSS_DH = 13;
    CspParameters cspParam = new CspParameters(PROV_DSS_DH);
    cspParam.KeyContainerName = null;
    cspParam.KeyNumber = (int)KeyNumber.Signature;
    cspParam.Flags = CspProviderFlags.UseDefaultKeyContainer;
    DSACryptoServiceProvider csp = new DSACryptoServiceProvider(cspParam);
    return csp.CspKeyContainerInfo.UniqueKeyContainerName;
}

the function returns the name of the file which will be created in the directory %APPDATA%\Microsoft\Crypto\DSS\[SID] and which will contain the generated key pair. 3) You logout the test account and login with an another account which has local administrative rights. You reset the password of the test account. 4) You login one more time with the test account and verify that the test program, which you posted, compiled in Visual Studio 2010 for .NET 4.0 produces the error NTE_BAD_KEY_STATE (0x8009000b) and the corresponding exception will be thrown. 5) If you recompile the program for .NET 3.5 instead of .NET 4.0 (you can use Visual Studio 2010 also) the test program will be run without any errors.

So all results will be exactly like there are described in the forum thread.

If you delete or renamed the file with the default key container for the DSA provider the problem will be solved. Like I described before after resetting of the users password the contain of the default key container will be unable to be decrypted. So if you know the name of the default container which is unique for the user (you can see the name for example from the traces of the Process Monitor) you can just copy any key container file from any other user and any other computer to the directory %APPDATA%\Microsoft\Crypto\DSS\[SID], rename the file so that the name will be the default container name and ... you will have absolutely the same results like with the resetting of the users password.

I made some experiments with different settings of the CspParameters as the parameter of DSACryptoServiceProvider (see my first suggestions about the usage of CspProviderFlags.CreateEphemeralKey), but without any success. After that I debugged the source code of .NET 4.0 and can definitively say that the call of _OpenCSP function, which code is not opened, will be called with the parameters which are independent from the parameters of the DSACryptoServiceProvider constructor. So one can not find a workaround of the problem for .NET 4.0 in the way with different settings of the CspParameters as the parameter of DSACryptoServiceProvider.

So if you want to modify your code so that it should work also in the situation with the corrupted default key provider I see currently only 2 ways to solve the problem:

1. Implement the whole or the part of code using unmanaged CryptAcquireContext function with the CRYPT_VERIFYCONTEXT flag.
2. Detect the problem with the error NTE_BAD_KEY_STATE (0x8009000b) and include the code part which delete or temporary renamed the file from %APPDATA%\Microsoft\Crypto\DSS\[SID] which contains the corrupted default key container. To detect the name of the file I think you can try to use CryptGetProvParam function with the PP_UNIQUE_CONTAINER or/and PP_ENUMCONTAINERS parameters.

I am sorry for the long text of my answer and thanks to all who are able to read it till this place. :-)

I hope that my answer do help you KristoferA to solve the problem and improve software which you develop.