Robocopy: ERROR 5 (0x00000005) Deleting Extra File \\location\to\my\file.txt Access is denied

I've created a Jenkins job which replicates files from source to destination. Both of them are network locations.

Here is the script that I use:

(IF NOT EXIST \\destination\files (net use \\destination\files /USER:"%my_user%" "%my_user_pwd%" ) ELSE (echo "mapped location already exists"))

(C:\Windows\System32\Robocopy.exe \\source\files \\destination\files /XD cache configurations util logs /NDL /S /COPY:DAT /MIR /R:2 /W:1 /MT:32) ^& IF %ERRORLEVEL% LEQ 8 exit /B 0

The job is copying new files which means user has write access to the destination. However it cannot delete any file from destination logging the following error:

ERROR 5 (0x00000005) Deleting Extra File \\desination\files\some_file.txt 
Access is denied.

Here "some_file.txt" exists in destination but source doesn't contain it so /MIR tries to delete it. However above error doesn't let it to remove the file.

I went through similar questions which didn't help that much. Please share any ideas that might be relevant. Thanks