Visual studio 2017 - debug works only for sites under DefaultAppPool

I have visual studio 17, on windows server 16, iis 10. When trying to run debug from visual studio I get this error:

Unable to start debugging on the web server. Operation not supported. Unknown error. 0x80004005

I have a customized application pool attached to the site.

.NET CLR version v4.0.30319
integrated
identity local system

I tried switching the version to 2.0, to classic , and enable and disable 32-bit app. There are some sites with the same problem whom has custom account identity.

The only thing that seems to disable the error is switching to DefaultAppPool. The problem is, the site is not working under DefaultAppPool. Also as I have about 30 sites on the server I can not set them all under DefaultAppPool.

Update: It's all about the Identity. The debugger only runs under ApplicationPoolIdentity. But if I set the app pool to ApplicationPoolIdentity I get permission errors on files.

How get I get the debugger run with custom identities?