System.Runtime.InteropServices.COMException (0x80004005) : The operation failed - Outlook VSTO Add-in trying to access Word

I have developed an outlook add-in, which creates a .docx file from the email body using VSTO tool in VS 2019. Using Office 365 on the windows RDS server, if I login as an admin and launch outlook and compose a mail, there is no issue the document gets created. But if I login as a non-admin, facing access issue of interop dlls.

The error gets logged when the below code is executed.

Microsoft.Office.Interop.Word.Document olDoc = _item.MailItem.GetInspector.WordEditor as Microsoft.Office.Interop.Word.Document;
tempAppFileName = tempFilePath + @"\" + _item.emailIdentifier.ToString() + ".docx";
olDoc.SaveAs2(tempAppFileName);

Getting the below error,

System.Runtime.InteropServices.COMException (0x80004005): The operation failed.
   at System.Dynamic.ComRuntimeHelpers.CheckThrowException(Int32 hresult, ExcepInfo& excepInfo, UInt32 argErr, String message)
   at CallSite.Target(Closure , CallSite , ComObject )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)

Tried providing access to the "GAC_MSIL" assembly folder, but it was of no use.

This is a strange issue, since there are 2 servers and it is getting replicated on one and not the other one.