Always Encrypted: 401 Errors From Azure Key Vault Is Cached Forever

Mar 12, 2025 by ADMIN 68 views

Introduction

In this article, we will discuss a critical issue that can occur when using Azure Key Vault to store keys for Always Encrypted in Azure SQL. The problem arises when a network configuration issue causes requests to the Key Vault to fail with an HTTP 401 error response. Even after the incident is resolved, the application still cannot read encrypted columns from Azure SQL due to the cached error response.

Describe the Bug

We use Azure Key Vault to store the necessary keys to decrypt data from Azure SQL (Always Encrypted). We had a network configuration issue several days ago in Azure making requests to our Azure Key Vault fail with an HTTP 401 error response. Even more than 24h after the incident was resolved, our application still couldn't read encrypted columns from Azure SQL.

Looking at the Date HTTP header shown in the exception message, we noticed that the Key Vault error response had been cached for more than 24h.

Error Details

Here is the details of an error that occurred on the 2025-03-04 around 12:00. Notice the date in the error (2025-03-03 around 09:00 AM).

Exception message:
Microsoft.Data.SqlClient.SqlException (0x80131904): Échec du déchiffrement de la colonne « [REDACTED] ».
Échec du déchiffrement d’une clé de chiffrement de colonne à l’aide du fournisseur de magasins de clés « AZURE_KEY_VAULT ». Vérifiez les propriétés de la clé de chiffrement de colonne et sa clé principale de colonne dans votre base de données. Les 10 derniers octets de la clé de chiffrement de colonne chiffrée sont : « [REDACTED] ».
Public network access is disabled and request is not from a trusted service nor via an approved private link.
Caller: [REDACTED]
Vault: [REDACTED]
Status: 403 (Forbidden)
ErrorCode: Forbidden

Content:
{"error":{"code":"Forbidden","message":"Public network access is disabled and request is not from a trusted service nor via an approved private link.\r\nCaller: [REDACTED]\r\nVault: [REDACTED]","innererror":{"code":"ForbiddenByConnection"}}}

Headers:
Cache-Control: no-cache
Pragma: no-cache
x-ms-keyvault-region: [REDACTED]
x-ms-client-request-id: [REDACTED]
x-ms-request-id: [REDACTED]
x-ms-keyvault-service-version: 1.9.2103.1
x-ms-keyvault-network-info: conn_type=Ipv4;addr=[REDACTED];act_addr_fam=InterNetwork;
x-ms-keyvault-rbac-assignment-id: REDACTED
X-Content-Type-Options: REDACTED
Strict-Transport-Security: REDACTED
Date: Mon, 03 Mar 2025 09:10:25 GMT    <<<<<<<<<<<<<<<<<<<<<<<<<<<<
Content-Type: application/json; charset=utf-8
Expires: -1
Content-Length: 713


Stack trace:
Microsoft.Data.SqlClient.SqlException:
   at Microsoft.Data.SqlClient.TdsParser.TryReadSqlValue (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadColumnInternal (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadColumn (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsync (Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__20.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=8.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__67`1.MoveNext (Microsoft.EntityFrameworkCore, Version=8.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__67`1.MoveNext (Microsoft.EntityFrameworkCore, Version=8.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [REDACTED]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [REDACTED]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [REDACTED]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [REDACTED]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0<br/>
**Always Encrypted: 401 Errors from Azure Key Vault is Cached Forever**
===========================================================

**Q&A**
------

**Q: What is the issue with Azure Key Vault and Always Encrypted?**
A: The issue arises when a network configuration issue causes requests to the Key Vault to fail with an HTTP 401 error response. Even after the incident is resolved, the application still cannot read encrypted columns from Azure SQL due to the cached error response.

**Q: What is the cause of the cached error response?**
A: The cached error response is caused by the `Microsoft.Data.SqlClient` library, which caches error responses for a very long period of time. This means that even after the incident is resolved, the application will still use the cached error response, preventing it from reading encrypted columns from Azure SQL.

**Q: How can I reproduce this issue?**
A: To reproduce this issue, you can configure an Azure SQL instance with Always Encrypted and store keys in a publicly available Azure Key Vault. Then, disable "public access" in the Key Vault config. As a result, a previously started app will still work, and a newly started app won't. If the public access is restored in the Key Vault configuration, the applications' behaviors won't change.

**Q: What is the expected behavior of `Microsoft.Data.SqlClient`?**
A: The expected behavior of `Microsoft.Data.SqlClient` is that it should not keep error responses in cache, or at least for a very limited period. This would prevent the application from using the cached error response and allow it to read encrypted columns from Azure SQL.

**Q: What are the technical details of this issue?**
A: The technical details of this issue are as follows:

* Microsoft.Data.SqlClient version: 5.2.2
* .NET target: .NET 8.0
* SQL Server version: Azure SQL
* Operating system: Docker container

**Q: How can I resolve this issue?**
A: To resolve this issue, you can try the following:

* Restart the entire application
* Update the `Microsoft.Data.SqlClient` library to a version that does not cache error responses for a very long period of time
* Use a different library or approach for handling error responses

**Q: Is this issue specific to Azure Key Vault and Always Encrypted?**
A: No, this issue is not specific to Azure Key Vault and Always Encrypted. It can occur with any library or approach that caches error responses for a very long period of time.

**Q: Can I prevent this issue from occurring in the future?**
A: Yes, you can prevent this issue from occurring in the future by:

* Using a library or approach that does not cache error responses for a very long period of time
* Implementing a mechanism to clear the cache after a certain period of time
* Using a different approach for handling error responses that does not rely on caching.