How to embed video in RenderTarget winforms in SharpDX

1

Forgive me. I'm just staring down the rabbit hole of DX. I'm using vb.net and SharpDX (hobby coder). I've managed to get a RenderTarget up and running and getting images rendered to it (extra class to one below). However, I'm totally stumped on how to embed videos onto the render target. My eventual aim is for a single window to display a mixture of images/text/videos via a layout editor.

The code below is scrappy, but should show how I've constructed the render target (successfully) but also how I've tried to incorporate the sharpdx repo demo here (which if I use as a standalone class, works).

It throws an exception in the render loop at:

mediaEngine.TransferVideoFrame(vidsurface, Nothing, New SharpDX.Rectangle(0, 0, 400, 300), Nothing)

Exception:

SharpDX.SharpDXException
  HResult=0x80004002
  Message=HRESULT: [0x80004002], Module: [General], ApiCode: [E_NOINTERFACE/No such interface supported], Message: No such interface supported

  Source=SharpDX
  StackTrace:
   at SharpDX.Result.CheckError()
   at SharpDX.MediaFoundation.MediaEngine.TransferVideoFrame(IUnknown dstSurfRef, Nullable`1 srcRef, RawRectangle dstRef, Nullable`1 borderClrRef)
   at Tests_SharpDX3.SharpDXRenderer._Closure$__50-0._Lambda$__0() in C:\Users\Steve\Documents\Visual Studio 2017\Projects\Tests_SharpDX3\Tests_SharpDX3\Classes\SharpDXRenderer.vb:line 297

(However, the sound plays for a bit!). Code:

Imports System.Windows.Forms
Imports SharpDX
Imports SharpDX.DXGI
Imports SharpDX.Direct3D
Imports SharpDX.Direct3D11
Imports SharpDX.Direct2D1
Imports SharpDX.Windows
Imports SharpDX.Mathematics
Imports SharpDX.MediaFoundation
Imports Device = SharpDX.Direct3D11.Device
Imports FactoryD2D = SharpDX.Direct2D1.Factory
Imports FactoryDXGI = SharpDX.DXGI.Factory1
Imports System.Threading

Public Class SharpDXRenderer


    ' **** Operational class level vars
    Dim device As Device
    Dim swapChain As SwapChain
    Dim lc As New LayoutConstructor

    'Video player vars
    Private Shared ReadOnly eventReadyToPlay As ManualResetEvent = New ManualResetEvent(False)
    Private Shared isMusicStopped As Boolean
    Private Shared mediaEngineEx As MediaEngineEx
    Private Shared dxgiManager As DXGIDeviceManager
    Public mediaEngine As MediaEngine
    Public vidDevice As Device
    Public vidsurface As Surface


    Public Sub New(lc As LayoutConstructor)

        _layoutConstructor = lc

    End Sub

    Public Sub CreateRenderTarget()

        ' Create render target window
        _renderWindow.Text = _renderWindowTitle
        _renderWindow.Height = _renderWindowHeight
        _renderWindow.Width = _renderWindowWidth

        ' Prevent window from being re-sized
        '_renderWindow.AutoSizeMode = AutoSizeMode.GrowAndShrink
        _renderWindow.AllowUserResizing = True


        ' Create swap chain description
        Dim swapChainDesc = New SwapChainDescription() With {
            .BufferCount = 2,
            .Usage = Usage.RenderTargetOutput,
            .OutputHandle = _renderWindow.Handle,
            .IsWindowed = _isWindowed,
            .ModeDescription = New ModeDescription(0, 0, New Rational(_refreshRate, 1), Format.R8G8B8A8_UNorm),
            .SampleDescription = New SampleDescription(1, 0),
            .Flags = SwapChainFlags.AllowModeSwitch,
            .SwapEffect = SwapEffect.Discard
        }

        ' Create swap chain And Direct3D device
        ' The BgraSupport flag Is needed for Direct2D compatibility otherwise RenderTarget.FromDXGI will fail!
        Device.CreateWithSwapChain(DriverType.Hardware, DeviceCreationFlags.BgraSupport, swapChainDesc, device, swapChain)

        ' Get back buffer in a Direct2D-compatible format (DXGI surface)
        Dim backBuffer As Surface = Surface.FromSwapChain(swapChain, 0)

        'Create Direct2D factory
        Using factory = New FactoryD2D()
            'Get desktop DPI
            Dim dpi = factory.DesktopDpi
            'Create bitmap render target from DXGI surface
            _renderTarget = New RenderTarget(factory, backBuffer, New RenderTargetProperties() With {
                .DpiX = dpi.Width,
                .DpiY = dpi.Height,
                .MinLevel = SharpDX.Direct2D1.FeatureLevel.Level_DEFAULT,
                .PixelFormat = New PixelFormat(Format.Unknown, Direct2D1.AlphaMode.Ignore),
                .Type = RenderTargetType.[Default],
                .Usage = RenderTargetUsage.None
            })
        End Using

        'Disable automatic ALT+Enter processing because it doesn't work properly with WinForms
        Using factory = swapChain.GetParent(Of FactoryDXGI)()
            factory.MakeWindowAssociation(_renderWindow.Handle, WindowAssociationFlags.IgnoreAltEnter)
        End Using

        ' Add event handler for ALT+Enter
        AddHandler _renderWindow.KeyDown, Sub(o, e)
                                              If e.Alt AndAlso e.KeyCode = Keys.Enter Then
                                                  swapChain.IsFullScreen = Not swapChain.IsFullScreen
                                              End If
                                          End Sub



        'Finally, populate the LayoutConstructor with the Render Target
        _layoutConstructor.RenderTarget = _renderTarget
        _layoutConstructor.RenderTargetSet = True

    End Sub

    Public Sub InitiliazeVideo()
        Dim mediafile As String = "Y:\System and Game Assets\Sony PlayStation\Games\Sony_Playstation\Video_MP4_HI_QUAL\3xtreme.mp4"

        MediaManager.Startup()
        'Dim renderForm = New SharpDX.Windows.RenderForm()
        'vidDevice = CreateDeviceForVideo(dxgiManager)
        Dim mediaEngineFactory = New MediaEngineClassFactory()

        Dim attr As MediaEngineAttributes = New MediaEngineAttributes()
        attr.VideoOutputFormat = CInt(SharpDX.DXGI.Format.B8G8R8A8_UNorm)
        attr.DxgiManager = dxgiManager

        mediaEngine = New MediaEngine(mediaEngineFactory, attr, MediaEngineCreateFlags.None)
        AddHandler mediaEngine.PlaybackEvent, AddressOf OnPlaybackCallback

        mediaEngineEx = mediaEngine.QueryInterface(Of MediaEngineEx)()

        Dim fileStream = System.IO.File.Open(mediafile, IO.FileMode.Open)
        Dim stream = New ByteStream(fileStream)
        Dim url = New Uri(mediafile, UriKind.RelativeOrAbsolute)
        mediaEngineEx.SetSourceFromByteStream(stream, url.AbsoluteUri)

        If Not eventReadyToPlay.WaitOne(1000) Then
            Console.WriteLine("Unexpected error: Unable to play this file")
        End If

        'swapChain = CreateSwapChain(device, renderForm.Handle)
        'Dim texture = Texture2D.FromSwapChain(Of Texture2D)(swapChain, 0)
        vidsurface = Surface.FromSwapChain(swapChain, 0)
        Dim w, h As Integer
        mediaEngine.GetNativeVideoSize(w, h)
        mediaEngineEx.Play()
        'Dim ts As Long
        'RenderLoop.Run(renderForm, Function()

        '                              If mediaEngine.OnVideoStreamTick(ts) Then
        '                                  mediaEngine.TransferVideoFrame(surface, Nothing, New SharpDX.Rectangle(0, 0, w, h), Nothing)
        '                              End If

        '                              swapChain.Present(1, SharpDX.DXGI.PresentFlags.None)
        '                          End Function)
        'mediaEngine.Shutdown()
        'swapChain.Dispose()
        'device.Dispose()
    End Sub

    Private Shared Sub OnPlaybackCallback(ByVal playEvent As MediaEngineEvent, ByVal param1 As Long, ByVal param2 As Integer)
        Select Case playEvent
            Case MediaEngineEvent.CanPlay
                eventReadyToPlay.[Set]()
            Case MediaEngineEvent.TimeUpdate
            Case MediaEngineEvent.[Error], MediaEngineEvent.Abort, MediaEngineEvent.Ended
                isMusicStopped = True
        End Select
    End Sub



    Public Sub RunRenderLoop()

        Dim clock = New System.Diagnostics.Stopwatch()
        Dim clockFrequency = CDbl(System.Diagnostics.Stopwatch.Frequency)
        clock.Start()
        Dim deltaTime = 0.0
        Dim fpsTimer = New System.Diagnostics.Stopwatch()
        fpsTimer.Start()
        Dim fps = 0.0
        Dim fpsFrames As Integer = 0

        Dim ts As Long ' linked with Video code

        RenderLoop.Run(_renderWindow, Function()
                                          _renderTarget.BeginDraw()
                                          _renderTarget.Transform = Matrix3x2.Identity
                                          _renderTarget.Clear(Color.DarkBlue)

                                          ' FPS display
                                          Dim totalSeconds = clock.ElapsedTicks / clockFrequency
                                          fpsFrames += 1
                                          If fpsTimer.ElapsedMilliseconds > 1000 Then
                                              fps = 1000 * fpsFrames / fpsTimer.ElapsedMilliseconds
                                              If _showFPS Then
                                                  ' Update window title with FPS once every second
                                                  _renderWindow.Text = String.Format("D3DRendering D3D11.1 - FPS: {0:F2} ({1:F2}ms/frame)", fps, CSng(fpsTimer.ElapsedMilliseconds) / fpsFrames)
                                              End If
                                              ' Restart the FPS counter
                                              fpsTimer.Reset()
                                              fpsTimer.Start()
                                              fpsFrames = 0
                                          End If

                                          'Draw the frame
                                          _layoutConstructor.DrawFrame()

                                          _renderTarget.EndDraw()

                                          'For Each mp In _mediaPlayers
                                          ' If mp.mediaEngine.OnVideoStreamTick(ts) Then
                                          '  mp.mediaEngine.TransferVideoFrame(mp.surface, Nothing, New SharpDX.Rectangle(0, 0, 400, 300), Nothing)
                                          ' End If
                                          'Next

                                          If mediaEngine.OnVideoStreamTick(ts) Then
                                              'Debug.WriteLine("tick")
                                              mediaEngine.TransferVideoFrame(vidsurface, Nothing, New SharpDX.Rectangle(0, 0, 400, 300), Nothing)
                                          End If

                                          swapChain.Present(0, PresentFlags.None)

                                          ' Determine the time it took to render the frame
                                          deltaTime = (clock.ElapsedTicks / clockFrequency) - totalSeconds

                                      End Function)

        _renderTarget.Dispose()
        swapChain.Dispose()
        device.Dispose()

    End Sub


End Class

Sadly, poor documentation and no SharpDX forums! :( Any Help appreciated.

vb.net
video
directx
sharpdx
asked on Stack Overflow Sep 26, 2018 by stigzler

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0