"Handle is invalid" error when opening SqlConnection

This error has started occurring sporadically and inexplicably, particularly when connecting to our session state database. Here's the error:

   Exception type: COMException 
    Exception message: The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()

A possibly related error appears at times in windows event viewer:

Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Threading.SemaphoreFullException
Stack:
   at System.Threading.Semaphore.Release(Int32)
   at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

EDIT: another flavor of the exception is as follows:

Exception Type: System.ComponentModel.Win32Exception
Error message: An operation was attempted on something that is not a socket
No Stack Trace Available
Exception Type: System.Data.SqlClient.SqlException
Error message: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An operation was attempted on something that is not a socket.)
at System.Data.SqlClient.TdsParser.TdsExecuteRPC(_SqlRPC[] rpcArray, Int32 timeout, Boolean inSchema, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj, Boolean isCommandProc, Boolean sync, TaskCompletionSource`1 completion, Int32 startRpc, Int32 startParam)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at System.Web.SessionState.SqlSessionStateStore.SqlExecuteReaderWithRetry(SqlCommand cmd, CommandBehavior cmdBehavior)
Exception Type: System.Web.HttpException
Error message: Unable to connect to SQL Server session database.
at System.Web.SessionState.SqlSessionStateStore.SqlExecuteReaderWithRetry(SqlCommand cmd, CommandBehavior cmdBehavior)
at System.Web.SessionState.SqlSessionStateStore.DoGet(HttpContext context, String id, Boolean getExclusive, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SqlSessionStateStore.GetItem(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Can anyone suggest:

1. What does this mean?
2. What could cause this (this is in an application that has be running for a long time very stably, with no major infrastructural changes happening before this began appearing)?
3. What might be done to resolve it?

As it turns out, we tracked the error down to deserializing a CancellationToken with Json.Net.

The underlying problem occurs when code is still trying to use an OS handle which has been freed. Of course, this can happen when your code works with handles directly. Our code does not do this, but it turns out that this can happen with Json.Net. Here's how:

We had a class as follows:

public class MyClass
{
   ...
}

// in one part of the code, this class was serialized & deserialized using Json.Net:
JsonConvert.SerializeObject(...);
JsonConvert.DeserializeObject<MyClass>(...);

The problem occurred when someone added a property to MyClass of type CancellationToken:

public class MyClass
{
    ...
    public CancellationToken Token { get; set; }
}

Here's the issue. When serialized, a CancellationToken looks like this:

{"IsCancellationRequested":false,"CanBeCanceled":true,"WaitHandle":{"Handle":{"value":1508},"SafeWaitHandle":{"IsInvalid":false,"IsClosed":false}}}

Note that doing so lazy-creates the token's WaitHandle property, and serializes the value of it underlying OS handle (1508).

When we deserialize the token, Json.Net will start with new CancellationToken() (equivalent to CancellationToken.None). It then will proceed to populate the Handle property of that token's WaitHandle using the saved IntPtr value. One obvious way in which this makes things go wrong is that the default CancellationToken's WaitHandle now points to a likely invalid handle. However, the bigger issue is that updating the handle dereferences the WaitHandle's original SafeHandle, thus allowing the garbage collector to run its finalizer and clean it up. You can then fall victim to the following set of events:

1. Handle 123 is allocated to a pooled database connection
2. A deserialization assigns handle 123 to the default cancellation token's WaitHandle
3. A second deserialization assigns a new handle value to the default cancellation token's WaitHandle
4. The garbage collector runs and finalizes the released 123 safe handle value
5. The database connection now points to an invalid handle

Here's some code which deliberately replicates the issue using a FileStream:

// serialize 2 tokens
var source = new CancellationTokenSource();
var serialized = JsonConvert.SerializeObject(source.Token);
var serialized2 = JsonConvert.SerializeObject(new CancellationTokenSource().Token);
var handle = source.Token.WaitHandle.Handle;
source.Dispose(); // releases source's handle

// spin until the OS gives us back that same handle as
// a file handle
FileStream fileStream;
while (true)
{
    fileStream = new FileStream(Path.GetTempFileName(), FileMode.OpenOrCreate);
    if (fileStream.Handle == handle) { break; }
}

// deserialize both tokens, thus releasing the conflicting handle
var deserialized = JsonConvert.DeserializeObject<CancellationToken>(serialized);
var deserialized2 = JsonConvert.DeserializeObject<CancellationToken>(serialized2);

GC.Collect();
GC.WaitForPendingFinalizers();

fileStream.WriteByte(1);
fileStream.Flush(); // fails with IOException "The handle is invalid"