One of the things we recently hit was authentication cookies and password reset tokens getting invalidated in an Azure App Service. The app uses deployment slots to deploy the new version there before swapping it to production. So what's common between authentication cookies and Identity password reset tokens in ASP.NET Core?

They both use Data Protection to encrypt their content. So what seemed to be happening is that those cookies/tokens were no longer accepted. The protection keys had changed.

What the issue is and why it happens

Let's start with a quote from the documentation on default settings for Data Protection:

If the app is hosted in Azure Apps, keys are persisted to the %HOME%\ASP.NET\DataProtection-Keys folder. This folder is backed by network storage and is synchronized across all machines hosting the app.

Keys aren't protected at rest.

The DataProtection-Keys folder supplies the key ring to all instances of an app in a single deployment slot.

Separate deployment slots, such as Staging and Production, don't share a key ring.

When you swap between deployment slots, for example swapping Staging to Production or using A/B testing, any app using Data Protection won't be able to decrypt stored data using the key ring inside the previous slot. This leads to users being logged out of an app that uses the standard ASP.NET Core cookie authentication, as it uses Data Protection to protect its cookies. If you desire slot-independent key rings, use an external key ring provider, such as Azure Blob Storage, Azure Key Vault, a SQL store, or Redis cache.

That last bit is the critical one. This is what happens:

  1. Version 1 of app published to production
  2. Data protection keys created and stored on the file system
  3. A user signs in, cookie created, encrypted with data protection keys
  4. Version 2 of app deployed to staging deployment slot
  5. Staging swapped to production
  6. The original data protection keys swapped to staging since they are in the file system
  7. New data protection keys generated and stored in production file system
  8. Version 2 app gets a request with the cookie
  9. Decrypting the cookie with data protection keys fails, data is invalid
  10. Authentication fails :(

Getting the keys out of the file system

To solve the issue, we need to get the data protection keys out of the file system. The keys are persisted to an XML file. So Azure Blob Storage works pretty well for that.

But the keys aren't encrypted by default. So anyone with access to the Storage account could access the keys used to secure authentication cookies etc. Not great.

And that's why we will additionally encrypt the keys using keys in Azure Key Vault.

Note that we mainly need to do this while running in Azure. This isn't usually necessary for local development.

Persisting keys to Azure Blob Storage

The first step we will take is to configure the keys to be stored in an Azure Storage account. So you'll need one of those if you're following along.

Now there are built-in functions to add Azure Blob Storage persistence for Data Protection, but none of those allowed for what we want. Namely we want to specify the blob reference factory passed to the AzureBlobXmlRepository instance. We want to use the Azure App Authentication library to leverage Managed Identities in Azure, and to leverage the developer's user account while developing locally (if you decide to do this while running locally as well).

Here is what we added in ConfigureServices of the Startup class:

var settings = Configuration.GetSection("DataProtection").Get<DataProtectionSettings>();

services.AddDataProtection();
// Replicates PersistKeysToAzureBlobStorage
// There is no overload to give it the func it ultimately uses
// We need to do that so that we can get refreshed tokens when needed
services.Configure<KeyManagementOptions>(options =>
{
    options.XmlRepository = new AzureBlobXmlRepository(() =>
    {
        // This func is called every time before getting the blob and before modifying the blob
        // Get access token for Storage
        // User / managed identity needs Blob Data Contributor on the Storage Account (container was not enough)
        string accessToken = _tokenProvider.GetAccessTokenAsync("https://storage.azure.com/", tenantId: settings.AadTenantId)
            .GetAwaiter()
            .GetResult();
        // Create blob reference with token
        var tokenCredential = new TokenCredential(accessToken);
        var storageCredentials = new StorageCredentials(tokenCredential);
        var uri = new Uri($"https://{settings.StorageAccountName}.blob.core.windows.net/{settings.StorageKeyContainerName}/{settings.StorageKeyBlobName}");
        // Note this func is expected to return a new instance on each call
        var blob = new CloudBlockBlob(uri, storageCredentials);
        return blob;
    });
});

The AzureServiceTokenProvider object is setup in the constructor:

private readonly AzureServiceTokenProvider _tokenProvider;

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
    _tokenProvider = new AzureServiceTokenProvider();
}

private IConfiguration Configuration { get; }

It's not very nice that the infrastructure for Data Protection is synchronous. Honestly it makes very little sense considering basically all key storage methods are asynchronous by nature.

Anyway, this is what the code does:

  1. Gets a strongly-typed object of configuration settings (you'll see what those look like in a bit)
  2. Adds services for Data Protection (so we can add Key Vault encryption to it in the next step)
  3. Configures the repository to use for storing keys
    • This is exactly what PersistKeysToAzureBlobStorage() would do if you called that on the object returned by AddDataProtection()
  4. In the function we:
    1. Get an Azure AD access token Storage through the Azure Service Authentication library
      • Note the user / service principal must have Blob Data Contributor role at the storage account level, for some reason setting it to container level did not work :/
    2. Builds a blob reference with the token and all the coordinates to locate the blob

Here is how the configuration looks like:

{
  "DataProtection": {
    "AadTenantId": "my-aad-tenant-id",
    "StorageAccountName": "mystorageaccountname",
    "StorageKeyContainerName": "dataprotection",
    "StorageKeyBlobName": "keys.xml"
  }
}

And the corresponding settings class:

public class DataProtectionSettings
{
    public string AadTenantId { get; set; }
    public string StorageAccountName { get; set; }
    public string StorageKeyContainerName { get; set; }
    public string StorageKeyBlobName { get; set; }
}

The Azure AD tenant id is needed only for local development. If you don't need it, you can remove the setting and remove its usage and leave the tenant id null.

With this in place, the keys would now be stored outside the app and everything would work great :)

Except for one little thing, the keys are not encrypted in Storage :(

Encrypting the encryption keys in Storage

To offer an additional layer of protection, we can encrypt the keys stored in Blob Storage using a key in Azure Key Vault. Setting it up is really easy.

First you'll of course need an Azure Key Vault. Then you can create a key in the vault.

Here's how you create a key:

  1. Open the Key Vault blade
  2. Go to Keys
  3. Click Generate/Import
  4. Give it a name
  5. Choose key type and key size
  6. Click Create

After creating, open the key and open the current version. You'll have the option to copy the key identifier, do that.

Add the key id to the config:

{
  "DataProtection": {
    "KeyVaultKeyId": "https://mykeyvaultname.vault.azure.net/keys/DataProtectionKey/bfc1bda979bc4081b89ab6f43bad12b8"
  }
}

And add a corresponding property to the settings class:

public class DataProtectionSettings
{
    public string KeyVaultKeyId { get; set; }
    // ... other properties
}

Then you'll need to create an access policy to allow the users/service principals to wrap and unwrap keys in the Key Vault.

Then we can quite easily add the Key Vault protection:

var kvClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(_tokenProvider.KeyVaultTokenCallback));

services.AddDataProtection()
    .ProtectKeysWithAzureKeyVault(kvClient, settings.KeyVaultKeyId);

And that's it. The keys file will now be encrypted/decrypted using the key in Key Vault. The key itself will never be in the app's memory.

Okay, we may have slightly over-simplified when saying the file is encrypted by Key Vault. What actually happens is that an AES key is generated to encrypt the XML file. The XML text is encrypted with the key. The key is then "wrapped" by using the Key Vault key. This wrapped key (i.e. encrypted key) is then stored alongside the encrypted data in Storage.

Then to decrypt the data, the wrapped key is unwrapped using Key Vault. Then that key is used to decrypt the data.

What this means is that the key in Key Vault is never in your app, and the Data Protection keys will never go to Key Vault. This is quite a good solution since it'll scale really well. Even if you had a ton of keys stored in the XML file, only the main generated encryption key needs to be wrapped/unwrapped with Key Vault.

You can see how the decryption works in the source code.

Final results

The libraries we have in the ASP.NET Core 2.1 app:

  • Microsoft.AspNetCore.App (no version specified)
  • Microsoft.AspNetCore.DataProtection.AzureKeyVault (2.1.1)
  • Microsoft.AspNetCore.DataProtection.AzureStorage (2.1.1)
  • Microsoft.AspNetCore.Razor.Design (2.1.2)
  • Microsoft.Azure.Services.AppAuthentication (1.2.0-preview2)
  • WindowsAzure.Storage (9.3.3)

The configuration:

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "AllowedHosts": "*",
  "DataProtection": {
    "KeyVaultKeyId": "https://mykeyvaultname.vault.azure.net/keys/DataProtectionKey/bfc1bda979bc4081b89ab6f43bad12b8",
    "AadTenantId": "my-azure-ad-tenant-id",
    "StorageAccountName": "mystorageaccountname",
    "StorageKeyContainerName": "dataprotection",
    "StorageKeyBlobName": "keys.xml"
  }
}

The AAD tenant id is only needed for local development, though there is no harm in specifying it otherwise.

Note every environment should use its own key file. So you'll want to make sure that the blob name/location is overridden in every environment.

The settings class corresponding to the config:

public class DataProtectionSettings
{
    public string KeyVaultKeyId { get; set; }
    public string AadTenantId { get; set; }
    public string StorageAccountName { get; set; }
    public string StorageKeyContainerName { get; set; }
    public string StorageKeyBlobName { get; set; }
}

Configuration in the Startup class:

private readonly AzureServiceTokenProvider _tokenProvider;

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
    _tokenProvider = new AzureServiceTokenProvider();
}

private IConfiguration Configuration { get; }

public void ConfigureServices(IServiceCollection services)
{
    services.Configure<CookiePolicyOptions>(options =>
    {
        // This lambda determines whether user consent for non-essential cookies is needed for a given request.
        options.CheckConsentNeeded = context => true;
        options.MinimumSameSitePolicy = SameSiteMode.None;
    });
    services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

    var settings = Configuration.GetSection("DataProtection").Get<DataProtectionSettings>();

    var kvClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(_tokenProvider.KeyVaultTokenCallback));

    services.AddDataProtection()
        .ProtectKeysWithAzureKeyVault(kvClient, settings.KeyVaultKeyId);
    // Replicates PersistKeysToAzureBlobStorage
    // There is no overload to give it the func it ultimately uses
    // We need to do that so that we can get refreshed tokens when needed
    services.Configure<KeyManagementOptions>(options =>
    {
        options.XmlRepository = new AzureBlobXmlRepository(() =>
        {
            // This func is called every time before getting the blob and before modifying the blob
            // Get access token for Storage
            // User / managed identity needs Blob Data Contributor on the Storage Account (container was not enough)
            string accessToken = _tokenProvider.GetAccessTokenAsync("https://storage.azure.com/", tenantId: settings.AadTenantId)
                .GetAwaiter()
                .GetResult();
            // Create blob reference with token
            var tokenCredential = new TokenCredential(accessToken);
            var storageCredentials = new StorageCredentials(tokenCredential);
            var uri = new Uri($"https://{settings.StorageAccountName}.blob.core.windows.net/{settings.StorageKeyContainerName}/{settings.StorageKeyBlobName}");
            // Note this func is expected to return a new instance on each call
            var blob = new CloudBlockBlob(uri, storageCredentials);
            return blob;
        });
    });
}

Some test code to see it all works:

public class HomeController : Controller
{
    private const string CookieName = "TestCookie";
    private readonly IDataProtector _dataProtector;

    public HomeController(IDataProtectionProvider dataProtectionProvider)
    {
        _dataProtector = dataProtectionProvider.CreateProtector("Test");
    }

    public IActionResult Index()
    {
        if (!Request.Cookies.TryGetValue(CookieName, out var cookieValue))
        {
            string valueToSetInCookie = $"Some text set in cookie at {DateTime.Now.ToString()}";
            var encryptedValue = _dataProtector.Protect(valueToSetInCookie);
            Response.Cookies.Append(CookieName, encryptedValue, new Microsoft.AspNetCore.Http.CookieOptions
            {
                IsEssential = true
            });
            return RedirectToAction("Index");
        }

        ViewBag.CookieValue = _dataProtector.Unprotect(cookieValue);
        return View();
    }
}

And the corresponding view:

@{
    ViewData["Title"] = "Home Page";
}

<h2>Decrypted value from cookie:</h2>
<p>@ViewBag.CookieValue</p>

With this test code we can confirm that the cookie value is successfully decrypted even after deploying a new version to a deployment slot and swapping that to production.

Summary

After configuring Azure Storage persistence and Key Vault protection, we now have a nice solution that works across deployment slot swaps. It also uses exactly zero credentials stored in the app to talk to both Azure Storage and Azure Key Vault.

Hope this is useful, until next time :)

Links