Not enough storage is available to process this command (IIS Worker Process memory leak)

I have 64bit Windows Server 2012 where I am getting error:

Message: Retrieving the COM class factory for component with CLSID {XXXX-XXXX-XXXX-XXXX} failed due to the following error: 80070008 Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008).

The server runs in a farm with another two 32 bit Windows 2003 servers with code built for 64 bit environment. This only happens on new 64 bit server, where the IIS worker process starts "eating up" the memory.

I have temporary resolved the problem with setting the application pool recycling in IIS. What could be the issue? Another info is that I am using 32 bit application pool, because one of the components is not build (yet) for 64 bit environment.

Any help regarding this leak would be appreciated!