Unable to read large log file with MemoryMappedViewStream

0

Question: Is there a way to read files in excess of 2GB using MemoryMappedViewStream?

I am attempting to read log files that are anywhere from 1 to 12 GB. The 1GB files read OK, but I am receiving the following error when reading a 4GB file:

System.IO.IOException HResult=0x80070008 Message=Not enough storage is available to process this command.

Source=System.Core StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 offset, Int64 size) at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewStream(Int64 offset, Int64 size, MemoryMappedFileAccess access) at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewStream() at ExchIISParserLib.LogParser.ParseLogs(Int32 daysago) in ...

My system has plenty of disk and memory space available to read the 4GB file. The line of code in question is:

MemoryMappedViewStream memoryMappedViewStream = MemoryMappedFile.CreateFromFile(log, FileMode.Open).CreateViewStream();

In my research efforts, I have found that MemoryMappedViewStream seems to be problematic when files exceed 2GB.

https://stackoverflow.com/a/49738895/4630376

I have looked at the offset and size parameters for the CreateViewStream() method. But those appear to just create a static window over the specified file, which does not read the entire file.

c#
memory
asked on Stack Overflow Dec 10, 2018 by J Weezy

1 Answer

0

A memory mapped view stream is a stream over a memory mapped view. It does not provide a stream on the whole file, but just the part that you map. You will still need to map the file in chunks to read the whole thing. Unless you really need shared memory you are probably just better off with reading the file in chunks.

answered on Stack Overflow Dec 10, 2018 by Mike

User contributions licensed under CC BY-SA 3.0