sendkeys.Send - Failing suddenly after being run many times - Error - Access Denied - RDP .net

.net Automated task is hosted in a server that I connect through RDP.

Sending keys to a windows application(Not web browser) SendKeys.send is working fine for a few runs and suddenly giving an error as Access denied - 0x80004005

When I login and then trigger the task it's working fine. Also when the server is idle task is performing well until a few runs.