How to fix photocapture error with vuforia, script in unity for hololens

I am trying to take a photo through the Hololens with my PhotoCapture unity script. I want to use a Vuforia Engine ARCamera for the possibility to see the reality at the same time as I see the AR-GUI I've created (for future functionality).

The main error I get is:

Failed capturing photo (hr = 0xC00D3704)

Why does it occur? How do I fix it?

The FocusManager singleton has not been initialized. UnityEngine.Debug:Assert(Boolean, String) HoloToolkit.Unity.Singleton`1:AssertIsInitialized() (at Assets/HoloToolkit/Common/Scripts/Singleton.cs:51) HoloToolkit.Unity.CanvasHelper:Start() (at Assets/HoloToolkit/Utilities/Scripts/CanvasHelper.cs:30)

is also an error occurring when starting the unity scene, but I haven't had it before...

This is code that I am using, placed on an ARCamera (have also tried out mixed reality camera with a vuforia behavior script and it didn't get the second error). Also, I want to apologize to the person who this code is borrowed from because I don't remember the link to your site.

public class PhotoCaptureExample : MonoBehaviour
{
    PhotoCapture photoCaptureObject = null;
    Texture2D targetTexture = null;
    public string path = "";
    CameraParameters cameraParameters = new CameraParameters();

void Start()
{

}

void Update()
{
    if (Input.GetKeyDown("k"))
    {
        Debug.Log("k was pressed");

        Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
        targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);

        // Create a PhotoCapture object
        PhotoCapture.CreateAsync(false, delegate (PhotoCapture captureObject)
        {
            photoCaptureObject = captureObject;
            cameraParameters.hologramOpacity = 0.0f;
            cameraParameters.cameraResolutionWidth = cameraResolution.width;
            cameraParameters.cameraResolutionHeight = cameraResolution.height;
            cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;

            // Activate the camera
            photoCaptureObject.StartPhotoModeAsync(cameraParameters, delegate (PhotoCapture.PhotoCaptureResult result)
            {
                // Take a picture
                photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
            });
        });
    }
}

string FileName(int width, int height)
{
    return string.Format("screen_{0}x{1}_{2}.png", width, height, DateTime.Now.ToString("yyyy-MM-dd_HH-mm-ss"));
}

void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    // Copy the raw image data into the target texture
    photoCaptureFrame.UploadImageDataToTexture(targetTexture);

    Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();

    targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
    targetTexture.Apply();

    byte[] bytes = targetTexture.EncodeToPNG();

    string filename = FileName(Convert.ToInt32(targetTexture.width), Convert.ToInt32(targetTexture.height));
    //save to folder under assets
    File.WriteAllBytes(Application.dataPath + "/Snapshots/" + filename, bytes);
    Debug.Log("The picture was uploaded");

    // Deactivate the camera
    photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}

void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
    // Shutdown the photo capture resource
    photoCaptureObject.Dispose();
    photoCaptureObject = null;
}
}

It seems that I can't get to OnCapturedPhotoToMemory or if it breaks already by the method call. Trying it out again right now the code occasionally won't register that I have pushed k at all...

Any help is appreciated!!

The problem is: Vuforia's VuforiaBehaviour on the camera holds the access on the device's real camera hardware => nothing else can use it meanwhile.

For fixing this you could use a dedicated camera for Vuforia (simply somewhere place a new GameObject e.g. VuforiaCamera in the scene and attach a Camera component as well as a VuforiaBehaviour to it.

On the Vuforia Camera set Culling Mask to Nothing (we don't render anything with that camera) and Depth to e.g. -2 (higher values are rendered on top -> put this behind all other cameras).

You have to do this because otherwise Vuforia automatically adds it to the main camera (which we don't want to disable because then we don't see anything). By manually adding one to the scene Vuforia automatically uses that one instead.

Everywhere in your scene where you need a Camera ofcourse you use the original camera from the Holo-Tookit (your usual MainCamera). Problem you can't fully rely on Camera.main in scripts because on runtime the VuforiaBehaviour automatically marks its Camera as MainCamera as well ... (-_-) Vuforia ... so additionally I always disabled and enabled the VuforiaBehaviour along with the GameObject but maybe it is enough already to only disable and enable the GameObject. At least at GameStart it should be disabled I guess until everything relying on Camera.main is finished.

Then you can simply disable that VuforiaCamera that has the VuforiaBehaviour on it.

VuforiaBehaviour.Instance.gameObject.SetActive(false);

// Note: that part I'm just inventing since I did it different
// using reference etc. I hope vuforia does not destroy the Instance
// OnDisable .. otherwise you would need the reference instead of using Instance here
// but maybe it is already enough to just enable and disable the GameObject
VuforiaBehaviour.Instance.enabled = false;

By doing this the device's camera was "free" and PhotoCapture can now use it.

PhotoCapture.StartPhotoModeAsync(....

If you then need the camera again for vuforia, first stop the PhotoCapture

PhotoCapture.StopPhotoModeAsync(..

and then after it stopped in the callback re-enable the ARCamera (object with the VuforiaBehaviour) and the VuforiaBehaviour again.

Something like

private void Awake()
{
    var cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
    targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);

    // Create a PhotoCapture object
    PhotoCapture.CreateAsync(false, captureObject =>
    {
        photoCaptureObject = captureObject;
        cameraParameters.hologramOpacity = 0.0f;
        cameraParameters.cameraResolutionWidth = cameraResolution.width;
        cameraParameters.cameraResolutionHeight = cameraResolution.height;
        cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
    });
}

private void Update()
{
    // if not initialized yet don't take input
    if (photoCaptureObject == null) return;

    if (Input.GetKeyDown("k"))
    {
        Debug.Log("k was pressed");

        VuforiaBehaviour.Instance.gameObject.SetActive(false);

        // Activate the camera
        photoCaptureObject.StartPhotoModeAsync(cameraParameters, result =>
        {
           if (result.success)
           {
               // Take a picture
               photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
           }
           else
           {
               Debug.LogError("Couldn't start photo mode!", this);
           }
       });
    }
}

private static string FileName(int width, int height)
{
    return $"screen_{width}x{height}_{DateTime.Now:yyyy-MM-dd_HH-mm-ss}.png";
}

private void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    // Copy the raw image data into the target texture
    photoCaptureFrame.UploadImageDataToTexture(targetTexture);

    Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();

    targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
    targetTexture.Apply();

    byte[] bytes = targetTexture.EncodeToPNG();

    string filename = FileName(Convert.ToInt32(targetTexture.width), Convert.ToInt32(targetTexture.height));
    //save to folder under assets
    File.WriteAllBytes(Application.streamingAssetsPath + "/Snapshots/" + filename, bytes);
    Debug.Log("The picture was uploaded");

    // Deactivate the camera
    photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}

private void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
    // Shutdown the photo capture resource
    photoCaptureObject.Dispose();
    photoCaptureObject = null;

    VuforiaBehaviour.Instance.gameObject.SetActive(true);
}

The exception might however also be related to this issue.