Video Recording Hangs on IMFSinkWriter->Finalize

I’ve implemented custom IMFMediaSink for use with sink writer. Works OK, receives h264 video samples. I don’t have any container, I’m consuming raw h264 video samples. I have not implemented custom writer, I'm using MFCreateSinkWriterFromMediaSink API to wrap my custom media sink into a framework-provided writer.

I’m unable to implement graceful shutdown, IMFSinkWriter::Finalize() never returns. When I implemented IMFSinkWriterCallback, IMFSinkWriter::Finalize() returns immediately but my IMFSinkWriterCallback::OnFinalize was never called.

The problem reproduces in 100% tests with both nvenc and MS software encoder.

Writer attributes:

MF_LOW_LATENCY = TRUE
MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS = TRUE (1)
MF_READWRITE_DISABLE_CONVERTERS = FALSE (2)
MF_SINK_WRITER_DISABLE_THROTTLING = TRUE
MF_SINK_WRITER_D3D_MANAGER
MF_SINK_WRITER_ASYNC_CALLBACK

(1) Tried both, same result

(2) Need the converters because nvenc only supports YUV and I have RGB textures on input.

Output media type (it’s fixed, I’m using the built-in handler created by MFCreateSimpleTypeHandler API).

MF_MT_MAJOR_TYPE = MFMediaType_Video
MF_MT_SUBTYPE = MFVideoFormat_H264
MF_MT_INTERLACE_MODE = MFVideoInterlace_Progressive
MF_MT_AVG_BITRATE = 40*1000*1000
MF_MT_FRAME_SIZE = { 3840, 2160 }
MF_MT_FRAME_RATE = { 60, 1 }
MF_MT_PIXEL_ASPECT_RATIO = { 1, 1 }

Input media type:

MF_MT_MAJOR_TYPE = MFMediaType_Video
MF_MT_SUBTYPE = MFVideoFormat_RGB32
MF_MT_INTERLACE_MODE = MFVideoInterlace_Progressive
MF_MT_FRAME_SIZE = { 3840, 2160 }
MF_MT_FRAME_RATE = { 60, 1 }
MF_MT_PIXEL_ASPECT_RATIO = { 1, 1 }

When not using IMFSinkWriterCallback, here’s a call stack at the time of hang:

ntdll.dll!_NtWaitForSingleObject@12 ()
KernelBase.dll!WaitForSingleObjectEx()
mfreadwrite.dll!CMFSinkWriter::InternalFinalize(void)
mfreadwrite.dll!CMFSinkWriter::Finalize(void)

MFTrace doesn’t have anything related to finalize even with -k All:

13700,3C60 19:01:25.79566 CMFTransformDetours::ProcessOutput @02EA6E3C failed hr=0xC00D6D72 MF_E_TRANSFORM_NEED_MORE_INPUT
13700,2A98 19:01:25.80250 CMFTransformDetours::ProcessOutput @1A6CEF38 Stream ID 0, Sample @1C244F30, Time 1216ms, Duration 16ms, Buffers 1, Size 12441600B, MFSampleExtension_CleanPoint=1;MFSampleExtension_Interlaced=0
13700,2098 19:01:25.80254 CMFTransformDetours::ProcessInput @02EA6E3C Stream ID 0, Sample @1C244F30, Time 1216ms, Duration 16ms, Buffers 1, Size 12441600B, MFSampleExtension_CleanPoint=1;MFSampleExtension_Interlaced=0
13700,2A98 19:01:25.80256 CMFTransformDetours::ProcessOutput @1A6CEF38 failed hr=0xC00D6D72 MF_E_TRANSFORM_NEED_MORE_INPUT
13700,2A98 19:01:25.80266 CMFTransformDetours::ProcessMessage @1A6CEF38 Message type=0x00000001 MFT_MESSAGE_COMMAND_DRAIN, param=00000000
13700,2A98 19:01:25.80267 CMFTransformDetours::ProcessOutput @1A6CEF38 failed hr=0xC00D6D72 MF_E_TRANSFORM_NEED_MORE_INPUT
13700,2098 19:01:25.81669 CMFTransformDetours::ProcessOutput @02EA6E3C Stream ID 0, Sample @1FB68CF8, Time 1216ms, Duration 16ms, Buffers 1, Size 680B, {2B5D5457-5547-4F07-B8C8-B4A3A9A1DAAC}=1;{73A954D4-09E2-4861-BEFC-94BD97C08E6E}=12166667 (0,12166667);{9154733F-E1BD-41BF-81D3-FCD918F71332}=65535;{973704E6-CD14-483C-8F20-C9FC0928BAD5}=1;MFSampleExtension_CleanPoint=0;{B2EFE478-F979-4C66-B95E-EE2B82C82F36}=16 (0,16)
13700,82C 19:01:25.81674 CMFTransformDetours::ProcessOutput @02EA6E3C failed hr=0xC00D6D72 MF_E_TRANSFORM_NEED_MORE_INPUT
13700,82C 19:01:25.81674 CMFTransformDetours::ProcessMessage @02EA6E3C Message type=0x00000001 MFT_MESSAGE_COMMAND_DRAIN, param=00000000
13700,82C 19:01:25.81674 CMFTransformDetours::ProcessOutput @02EA6E3C failed hr=0xC00D6D72 MF_E_TRANSFORM_NEED_MORE_INPUT
13700,1F54 19:01:27.24237 CKernel32ExportDetours::OutputDebugStringA @ D3D11 WARNING: Process is terminating. Using simple reporting. Please call ReportLiveObjects() at runtime for standard reporting. [ STATE_CREATION WARNING #0: UNKNOWN] 
13700,1F54 19:01:27.24255 CKernel32ExportDetours::OutputDebugStringA @ D3D11 WARNING: Live Producer at 0x0311D91C, Refcount: 13. [ STATE_CREATION WARNING #0: UNKNOWN] 

Warnings about live D3D resources are expected as I terminated the process after the hang.

Any ideas what’s going on? I think the writer probably waits for these SPS/PPS magic blobs to arrive but it never happens. Is there a way to instruct the h264 encoder to output SPS/PPS somewhere?

You’ve implemented custom IMFMediaSink, so i suppose you 've also implemented IMFStreamSink.

Doing this in the usual way with Mediafoundation, you have circular COM reference between IMFMediaSink and IMFStreamSink. That's why the Shutdown method from IMFMediaSink interface exists.

If a program that uses your custom MediaSink does not call Shutdown at the right place, there will be memory leaks.

From your IMFSinkWriterCallback problem, we don't have enough information to find where the problem is.

Also it is not clear about "custom IMFMediaSink" and "IMFSinkWriter". Are you also implementing a IMFSinkWriter...

EDIT1

Just two things :

MFCreateSinkWriterFromMediaSink

Call CoInitialize(Ex) and MFStartup before calling this function.

When you are done using the media sink, call the media sink's IMFMediaSink::Shutdown method. (The sink writer does not shut down the media sink.) Release the sink writer before calling Shutdown on the media sink.

do you release interfaces correctly ?

IMFSinkWriter::Finalize

Internally, this method calls IMFStreamSink::PlaceMarker to place end-of-segment markers for each stream on the media sink.

Do you handle this message (MFSTREAMSINK_MARKER_ENDOFSEGMENT) ?

We don't know how you handle CriticalSection/Event/CircularReference, so it's hard to found the problem.

EDIT2

Is there a way to instruct the h264 encoder to output SPS/PPS somewhere?

Normally, for h264 video format, you need to get the attributes MF_MT_MPEG_SEQUENCE_HEADER (BLOB type) when SetCurrentMediaType is called on your IMFStreamSink (assuming you implements IMFMediaTypeHandler).

EDIT3

Could you provide the real one (this is how i think the app should be) :

I don't remember if your custom sink creates a mp4 file. If it is, in the IMFSinkWriter::Finalize, you have to generate ftpy/moov atom.

EDIT4

Also you can read this : Video Recording Hangs on IMFSinkWriter->Finalize();

With no source code, this is the only answer i can give.