Cannot add Smart Card Certificate to Yubikey

I am trying to create a smartcard certificate and add it back to the Yubikey (I am using Yubico's Mini driver so the yubikey behaves like a smartcard and cannot use their PIVManager or YKMan). I am able to successfully sign the CSR with the yubikey with the following code:

certificateRequest.CertRequest = new CX509CertificateRequestPkcs10();
certificateRequest.CertRequest.Initialize(X509CertificateEnrollmentContext.ContextUser);
certificateRequest.CertRequest.PrivateKey.ExportPolicy = X509PrivateKeyExportFlags.XCN_NCRYPT_ALLOW_EXPORT_NONE;
certificateRequest.CertRequest.PrivateKey.Length = 2048;
certificateRequest.CertRequest.PrivateKey.ProviderName = "Microsoft Smart Card Key Storage Provider";
certificateRequest.CertRequest.PrivateKey.KeyUsage = X509PrivateKeyUsageFlags.XCN_NCRYPT_ALLOW_SIGNING_FLAG;
certificateRequest.CertRequest.PrivateKey.KeySpec = X509KeySpec.XCN_AT_NONE;
certificateRequest.CertRequest.PrivateKey.MachineContext = false;
var subjectEncoded = new CX500DistinguishedNameClass();
subjectEncoded.Encode(certificateRequest.SubjectName);
certificateRequest.CertRequest.Subject = subjectEncoded;
certificateRequest.CertRequest.Encode();
certificateRequest.CSR = certificateRequest.CertRequest.RawData[EncodingType.XCN_CRYPT_STRING_BASE64REQUESTHEADER];

then I go to the CA and get the certificate back. When I try to to add the certificate back to the Yubikey i get the following error:

CertEnroll::CX509Enrollment::InstallResponse: Cannot find object or property. 0x80092004 (-2146885628 CRYPT_E_NOT_FOUND)

which according with what I found on google it means that the System cannot find the private key with which the certificate was signed. I am using the request to initialize the container and it still cant find that it was done by a smartcard, here is the code for reference:

CX509Enrollment objEnroll = new CX509EnrollmentClass();
objEnroll.InitializeFromRequest(certificateRequest.CertRequest);
objEnroll.InstallResponse(
    InstallResponseRestrictionFlags.AllowUntrustedRoot,
    certificateRequest.StringCert,
    EncodingType.XCN_CRYPT_STRING_BASE64,
    null
);

Is there a way to tell windows to look for the private key in the YubiKey?

I was missing the actual creation of the private Key and of the request here is the new complete code:

certificateRequest.CertRequest = new CX509CertificateRequestPkcs10();
certificateRequest.CertRequest.Initialize(X509CertificateEnrollmentContext.ContextUser);
certificateRequest.CertRequest.PrivateKey.ExportPolicy = X509PrivateKeyExportFlags.XCN_NCRYPT_ALLOW_EXPORT_NONE;
certificateRequest.CertRequest.PrivateKey.Length = 2048;
certificateRequest.CertRequest.PrivateKey.ProviderName = "Microsoft Smart Card Key Storage Provider";
certificateRequest.CertRequest.PrivateKey.KeyUsage = X509PrivateKeyUsageFlags.XCN_NCRYPT_ALLOW_SIGNING_FLAG;
certificateRequest.CertRequest.PrivateKey.KeySpec = X509KeySpec.XCN_AT_NONE;
certificateRequest.CertRequest.PrivateKey.MachineContext = false;
certificateRequest.CertRequest.PrivateKey.Create();
var subjectEncoded = new CX500DistinguishedNameClass();
subjectEncoded.Encode(certificateRequest.SubjectName);
certificateRequest.CertRequest.Subject = subjectEncoded;
certificateRequest.CertRequest.Encode();
certificateRequest.CSR = certificateRequest.CertRequest.RawData[EncodingType.XCN_CRYPT_STRING_BASE64REQUESTHEADER];

then I go to the CA and get the certificate back. When I try to to add the certificate back to the Yubikey:

CX509Enrollment objEnroll = new CX509EnrollmentClass();
objEnroll.InitializeFromRequest(certificateRequest.CertRequest);
objEnroll.CreateRequest(EncodingType.XCN_CRYPT_STRING_BASE64);
objEnroll.InstallResponse(
    InstallResponseRestrictionFlags.AllowUntrustedRoot,
    certificateRequest.StringCert,
    EncodingType.XCN_CRYPT_STRING_BASE64,
    null
);