"COM object separated from its underlying RCW" error when using VirtualFileDataObject with async/await

I'm trying to use David Anson's VirtualFileDataObject (as shipped in this NuGet package), and there's a few problems I'm running into. One occurs when passing an async void to the StreamContents property. This will work fine:

Dim descriptor As New VirtualFileDataObject.VirtualFileDataObject.FileDescriptor With {
    .Name = info.FileName,
    .StreamContents = Sub(s)
                          Dim d As New FileTransfer.Downloader(info.BildID)
                          Dim fs = d.DownloadSingleAsync().GetAwaiter().GetResult()

                          fs.Position = 0

                          fs.CopyTo(s)
                      End Sub}

(Downloader is a class that fetches data using web services. It shouldn't be relevant, as DownloadSingleAsync returns a TaskStream.)

But of course, GetAwaiter() isn't how you're supposed to await something.

This, however:

Dim descriptor As New VirtualFileDataObject.VirtualFileDataObject.FileDescriptor With {
    .Name = info.FileName,
    .StreamContents = Async Sub(s)
                          Dim d As New FileTransfer.Downloader(info.BildID)
                          Dim fs = Await d.DownloadSingleAsync().ConfigureAwait(False)

                          fs.Position = 0

                          Await fs.CopyToAsync(s).ConfigureAwait(False)
                      End Sub}

…will throw a COMException.

My guess is that something is released too soon. I found this piece of code:

// Return an IntPtr for the IStream
ptr = Marshal.GetComInterfaceForObject(iStream, typeof(IStream));
Marshal.ReleaseComObject(iStream);

Perhaps ReleaseComObject is called too soon. Is there some call I can make (perhaps similar to GC.SuppressFinalize()?) to prevent the COM object from being released until I'm done?

(edit) As requested, a stacktrace:

System.Runtime.InteropServices.InvalidComObjectException
  HResult=0x80131527
  Message=COM object that has been separated from its underlying RCW cannot be used.
  Source=mscorlib
  StackTrace:
   at System.StubHelpers.StubHelpers.StubRegisterRCW(Object pThis)
   at System.Runtime.InteropServices.ComTypes.IStream.Write(Byte[] pv, Int32 cb, IntPtr pcbWritten)
   at System.IO.Stream.<>c.<BeginWriteInternal>b__46_0(Object )
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.Stream.EndWrite(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at MyApp.Bilder._Closure$__178-0.VB$StateMachine___Lambda$__0.MoveNext() in Bilder.vb:line 467