UWP AudioGraph - Attempted to read or write protected memory

I am trying to use the DSharpPlus library and the UWP AudioGraph API to write raw PCM bytes to an AudioFrameInputNode, however I keep getting memory access violations: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. and Exception thrown at 0x00007FFE447ED540 (ntdll.dll) in xyz.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.

I also get: Unhandled exception at 0x00007FFE447EF51A (ntdll.dll) in xyz.exe: RangeChecks instrumentation code detected an out of range array access.

My code is:

private void CreateAudioFrameInputNode(ref AudioEncodingProperties properties, VoiceReceiveEventArgs sender, ref AudioFrameInputNode inputNode)
{
    properties = AudioEncodingProperties.CreatePcm((uint)sender.AudioFormat.SampleRate, (uint)sender.AudioFormat.ChannelCount, 16);
    inputNode = App.AudioGraph.CreateFrameInputNode(properties);
    inputNode.AddOutgoingConnection(App.OutputNode);
    inputNode.Start();
    App.InputNodes.Add(sender.SSRC, inputNode);
}

....

   lock (_outputLock)
    {
        AudioFrameInputNode inputNode = null;
        AudioEncodingProperties properties = null;

        if (!App.InputNodes.ContainsKey(e.SSRC))
        {
            CreateAudioFrameInputNode(ref properties, e, ref inputNode);
        }
        else
        {
            inputNode = App.InputNodes[e.SSRC];
            properties = inputNode.EncodingProperties;

            if (properties.ChannelCount != e.AudioFormat.ChannelCount)
            {
                //??
            }
        }

        try
        {
            var frame = new AudioFrame((uint)e.PcmData.Length);
            using (AudioBuffer buffer = frame.LockBuffer(AudioBufferAccessMode.Write))
            {
                using (IMemoryBufferReference reference = buffer.CreateReference())
                {
                    byte* dataInBytes;
                    uint capacityInBytes;
                    ((IMemoryBufferByteAccess)reference).GetBuffer(out dataInBytes, out capacityInBytes);
                    //Debug.Assert(capacityInBytes >= (uint)e.PcmData.Length * sizeof(short));

                    short* dataInShort = (short*)dataInBytes;
                    for (int i = 0; i < e.PcmData.ToArray().Length; i++)
                    {
                        dataInShort[i] = e.PcmData.ToArray()[i];
                    }
                }
            }
            inputNode.AddFrame(frame);
        }
        catch (Exception ex)
        {
            inputNode.DiscardQueuedFrames();
        }
    }

And the error is triggered on this line: dataInShort[i] = e.PcmData.ToArray()[i];

What am I doing wrong?