Azure Sql Column Level Encryption Error - Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'

I am working POC of API to store/retrieve some sensitive information to/from SQL database. API uses EF core for DB operations. I have configured the column level encryption with Azure KeyVault provider. Master key is getting generated in KeyVault.

EF Configuration Reference

https://entityframeworkcore.com/knowledge-base/60420382/ef-core-decryption-of-sql-column-level-encryption

I need to use managed identity to access encryption MasterKey form KeyVault. Below is the code to initialize KeyVault Provider and get token token using the Microsoft.Azure.Services.AppAuthentication library to access KeyVault.

private static void InitializeAzureKeyVaultProvider()
    {
        SqlColumnEncryptionAzureKeyVaultProvider azureKeyVaultProvider =
          new SqlColumnEncryptionAzureKeyVaultProvider(GetToken);

        Dictionary<string, SqlColumnEncryptionKeyStoreProvider> providers =
          new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>();

        providers.Add(SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, azureKeyVaultProvider);
        SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providers);
    }

    private static async Task<string> GetToken(string authority, string resource, string scope)
    {
        var azureServiceTokenProvider = new AzureServiceTokenProvider();
        string accessToken = await azureServiceTokenProvider.GetAccessTokenAsync("https://management.azure.com/").ConfigureAwait(false);
        return accessToken;
    }

Access token is getting generated successfully.

In addition to master key, I am storing connection string in KeyVault and configured KeyVault provider using Managed identity to retrieve connection string from KeyVault.

https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1

I deployed the Api App in Azure and enabled the Identity. I used this system generated ObjectId to add KeyVault access policy to allow Api to access KeyVault. I have provided the required permissions to keys such as get,wrapKey,unwrapKey,sign,verify,list

Now I can access the connection string and able connect to database. But when I am trying to save the record in database I am getting below error.

Looks like with a generated token I am not able to access the KeyVault. Am I missing anything?. Please help.

Microsoft.EntityFrameworkCore.DbUpdateException HResult=0x80131500 Message=An error occurred while updating the entries. See the inner exception for details. Source=Microsoft.EntityFrameworkCore.Relational StackTrace: at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable1 commandBatches, IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChanges(IList1 entries) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList1 entriesToSave) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(DbContext _, Boolean acceptAllChangesOnSuccess) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func`3 verifySucceeded) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess) at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess) at Microsoft.EntityFrameworkCore.DbContext.SaveChanges() at DbEncryptionApi.Controllers.MedicalController.SavePatientRecord(PatientDto patient) in D:\Data\Projects\DbEncryptionApi\DbEncryptionApi\Controllers\MedicalController.cs:line 34 at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()

This exception was originally thrown at this call stack: [External Code]

Inner Exception 1: SqlException: Failed to decrypt a column encryption key using key store provider: 'AZURE_KEY_VAULT'. Verify the properties of the column encryption key and its column master key in your database. The last 10 bytes of the encrypted column encryption key are: 'C6-C8-F6-58-A0-DE-6F-68-73-9F'. One or more errors occurred. (Operation returned an invalid status code 'Unauthorized')

Inner Exception 2: KeyVaultErrorException: Operation returned an invalid status code 'Unauthorized'

From the error it looks like you might not have the column encryption key properly configured and linked to the master key in KeyVault.

I suggest you review the setup of the column encryption in the database - perhaps using the wizard to verify the setup to KeyVault as a first step.