Very weird SSL error in .NET: The specified data could not be decrypted only for a specific URL

I'm using .NET to download data from a URL. For most URLs it works no problem, but for one specific URL, I am getting a very weird error when I try to make the connection. Furthermore, the error only happens on the 2nd (and subsequent) attempts to make the request. The first time always seems to work.

Here is some sample code which demonstrates the problem:

string url = "https://health-infobase.canada.ca/src/data/covidLive/covid19.csv";

for (int i = 1; i <= 10; i++)
{
    var req = (HttpWebRequest)WebRequest.Create(url);

    // Just in case, rule these out as being related to the issue.
    req.AllowAutoRedirect = false;
    req.ServerCertificateValidationCallback = (object s, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) => true;

    try
    {
        // This line throws the exception.
        using (req.GetResponse()) { }
    }
    catch (Exception ex) {
        Console.WriteLine(ex.ToString());
        Console.WriteLine($"Failed on attempt {i}.");
        return;
    }
}

Notes:

• Using any other URL other than the specified one seems to work. Even other URLs on the same server (with the same certificate) work without any trouble. E.g., https://health-infobase.canada.ca/pass.
• I turned the SChannel logging level up to 3 (warnings and errors), but didn't see anything from the SChannel source in the Windows event log.
• The problem happens both in .NET 4.8 (528372) as well as .NET Core 3.1.7
• The problem happens with both WebRequest and WebClient
• In .NET Framework 4.8, the problem seems to go away when using WebClient.DownloadData(), but it still occurs when using WebClient.OpenRead()
• In .NET Framework 4.8, the problem only seems to happen for URLs which download some file (like the one in my code example). In .NET Core however, the error happens for any URL with a path under https://health-infobase.canada.ca/src/.
• If I use an intermediary HTTPS sniffer (like Fiddler), then the problem goes away.
• Running the same code on .NET Core on Linux does not exhibit any problems.
• Using a hardwired certificate validation callback (ServicePointManager.ServerCertificateValidationCallback always returns true) does not help.

The stack trace when I run in .NET Core looks like this:

System.Net.WebException: The SSL connection could not be established, see inner exception. Authentication failed, see inner exception.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090330): The specified data could not be decrypted.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.Net.HttpWebRequest.SendRequest()
   at System.Net.HttpWebRequest.GetResponse()
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at UserQuery.Main() in C:\Users\robs\AppData\Local\Temp\LINQPad6\_gifldqtg\xltrxu\LINQPadQuery:line 12

On .NET Framework, the stack trace seems to be much less useful:

System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.GetResponse()
   at UserQuery.Main() in C:\Users\robs\AppData\Local\Temp\LINQPad5\_psduzptv\dcrjhq\LINQPadQuery.cs:line 48

Update: Submitted as an issue on github: https://github.com/dotnet/runtime/issues/43682

Update

I dug in some more and tried to forcefully close the SSL connection to the server. Sadly, there is no API or I just didn't find one. So we'll play with reflection a little bit.

for (int i = 1; i <= 25; i++)
{
    try
    {
        Console.WriteLine(i);

        // use new instances everytime now
        using var handler = new HttpClientHandler();
        using var client = new HttpClient(handler);

        // I found the stream contains a reference to the connection...
        await using var test = await client.GetStreamAsync(url);

        var connectionField = test.GetType()
            .GetField("_connection", BindingFlags.Instance | BindingFlags.NonPublic);
        var connection = connectionField.GetValue(test);

        // ...which contains a reference to the SslStream...
        var sslStreamField = connection.GetType()
            .GetField("_stream", BindingFlags.Instance | BindingFlags.NonPublic);
        var sslStream = sslStreamField.GetValue(connection) as SslStream;


        using var sr = new StreamReader(test);
        var data = await sr.ReadToEndAsync();
        Console.WriteLine(data.Substring(0, 100));

        // ... which I'll shutdown now.
        sslStream.Close();
        sslStream.Dispose();

        await Task.Delay(1000);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Failed on attempt {i} : {ex.Message}.");
    }
}

This works fine and we see our friend FIN, ACK in Wireshark. Interestingly the error still occurs in an alternating pattern. Maybe there is some broken load balancer or something going on - I don't know.

Here's the output

1
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
2
Failed on attempt 2 : The SSL connection could not be established, see inner exception..
3
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
4
Failed on attempt 4 : The SSL connection could not be established, see inner exception..
5
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
6
Failed on attempt 6 : The SSL connection could not be established, see inner exception..
7
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
8
Failed on attempt 8 : The SSL connection could not be established, see inner exception..
9
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
10
Failed on attempt 10 : The SSL connection could not be established, see inner exception..
11
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
12
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
13
Failed on attempt 13 : The SSL connection could not be established, see inner exception..
14
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
15
Failed on attempt 15 : The SSL connection could not be established, see inner exception..
16
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
17
Failed on attempt 17 : The SSL connection could not be established, see inner exception..
18
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra
19
Failed on attempt 19 : The SSL connection could not be established, see inner exception..
20
pruid,prname,prnameFR,date,numconf,numprob,numdeaths,numtotal,numtested,numrecover,percentrecover,ra

Old Answer (better IMO)

Not if sure what the question is here but well, here's a fix for you.

    static string url = "https://health-infobase.canada.ca/src/data/covidLive/covid19.csv";

    static async Task Fix1()
    {
        var client = new HttpClient();
        for (int i = 1; i <= 25; i++)
        {
            var response = await client.GetAsync(url);
            // .. do something
        }
    }

    static async Task Fix2()
    {
        var handler = new HttpClientHandler();
        for (int i = 1; i <= 25; i++)
        {
            var client = new HttpClient(handler);
            var response = await client.GetAsync(url);
            // .. do something
        }
    }

It seems weird indeed. Keeping the client or the handler around will keep the connection alive and prevent another key exchange.

Let's modify your example again and send a UDP trigger packet so we can see what happens after GetResponse.

static async Task WhatIsGoingOn()
{
    UdpClient udp = new UdpClient();
    for (int i = 1; i <= 25; i++)
    {
        var req = (HttpWebRequest)WebRequest.CreateHttp(url);
        using var data = req.GetResponse();

        await udp.SendAsync(new byte[] { 1, 2, 3, 4 }, 4, IPEndPoint.Parse("255.255.255.255"));
        GC.Collect();
        GC.WaitForFullGCComplete();
        GC.WaitForPendingFinalizers();
        await Task.Delay(200);
    }
}

In Wireshark we will se that the transfer is still ongoing when the trigger packet is sent. Another TLS connection is initiated gets reset by the server. Maybe rate limiting or limiting active connections to the server?

If you take your original loop and throttle with sleeps / Task.Delays you will get 3/4 or maybe more "successful" connections.

The fact that the connection persists even if the block scope is lost and the using should do its job or if Dispose is called manually is actually very weird.