Currently, we're trying to combine the use of Unity3D holograms and HTML rendering in one app. The idea works as follows:
1) start a XAML app whose main window contains a WebView component (using the XAML app template)
2) Switch to Holographic (DirectX) mode (essentially copy over the code from the Holographic app template)
3) The WebView still runs in the background
4) We can call the
5) Render the contents of the WebView via the
CapturePreviewToStreamAsync() API to a bitmap
6) Convert the bitmap to a texture and display it on the DirectX side- thus having HTML rendering with holograms
Our experiments so far work in:
However, we have problems if we try to do the same thing with Unity3D. Here is what we do:
1) We create a Unity3D project with mixed reality support
2) We export the Unity project with "UWP Build type" set to "XAML" (instead of D3D). That app builds within MSVS 2017 and starts as a XAML app which automatically switches to holographic space, showing correctly the scenes as defined in Unity.
3) We further edit the main XAML page to contain a WebView element (pointing to a webpage).
Now if we call the
CapturePreviewToStreamAsync() API of the WebView component, a
System.Exception is thrown with message:
Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))
The error is 100% reproducible and occurs if we use both the Hololens glasses and the emulator.
Any suggestions what might be the cause of the behavior? Seems that Unity creates the main window of the XAML part of the app in a different way, but we don't have access to that since the window creation happens mainly within the re-compiled Unity modules.
Although we couldn't find the reason for the exception, we found a workaround, by following the instructions in this blog entry by Jonathan Antoine :
The way to make things work was to:
1. Create a new XAML page for the Unity3D generated app from above (let's name it WebViewPage, which contains our WebView component)
2. Modify the Unity generated XAML app to navigate to WebViewPage on startup instead of to the MainPage
3. When needed to switch to holographic space, just navigate to the MainPage the usual way:
var frame = new Frame();
Window.Current.Content = frame;
4. You might also need to move the
appCallbacks object from the generated app from the
App class to the
That solved the problem. Now the
CapturePreviewToStreamAsync() call works and returns the data.
NOTE: We noticed that the app leaves the HoloLens emulator (10.0.14393.1358) in some state which doesn't allow the app to be launched a second time without restarting the emulator. But that problem doesn't show up with on a real device
Hope that helps someone
User contributions licensed under CC BY-SA 3.0