Unwanted memory consumption of CGImage draw calls

I'm working on a project that decodes an image into a RGBA software buffer using Core Graphics.

func LoadImage(imageFile: CFString, imageData: inout Array<UInt32>) {
        let imgUrl: CFURL = CFURLCreateWithFileSystemPath(nil, imageFile, CFURLPathStyle.cfurlposixPathStyle, false);
        let imgSource: CGImageSource = CGImageSourceCreateWithURL(imgUrl, nil)!;
        let imgProperties: NSDictionary = CGImageSourceCopyPropertiesAtIndex(imgSource, 0, nil)!;
        let pixelWidth: Int = (imgProperties.object(forKey: kCGImagePropertyPixelWidth) as! NSNumber).intValue;
        let pixelHeight: Int = (imgProperties.object(forKey: kCGImagePropertyPixelHeight) as! NSNumber).intValue;
        let img: CGImage = CGImageSourceCreateImageAtIndex(imgSource, 0, nil)!;
        let colorspace = CGColorSpaceCreateDeviceRGB();

        imageData = Array<UInt32>(repeating: 0x00000000, count: pixelWidth * pixelHeight);

        let ctx: CGContext = CGContext(data: &imageData[0], width: pixelWidth, height: pixelHeight, bitsPerComponent: 8, bytesPerRow: pixelWidth * 4, space: colorspace, bitmapInfo: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)!;

        ctx.draw(img, in: CGRect(x: 0, y: 0, width: pixelWidth, height: pixelHeight));
    }

When calling this function, several times (even with the same image file path) memory consumption quickly raises. Using a 7600x5066 JPEG I quickly get 2+ GB calling it several times with a delay of only milliseconds. Activity monitor shows an increase of real mem and purgable mem as soon as the draw call returns. Both real mem and purgeable mem keep increasing up to about 2 GB but seemingly not beyond.

Asuming that purgeable mem is part of real mem (aka the process' virtual memory). I'd expect that the system does what "purgeable" says, i.e. purging it at a given time. It does so seemingly as the amount never went beyond 2 GB, sometimes dropped to about 1.5.

Since I'm restricted to a 32 bit environment for now this is critical. Is there a way to prevent that extra allocation that ends up purgeable? If not, is there a way to force purging it?