Windows IFS: own filesystem VS notepad.exe

Windows 10 x64 - implemented own filesystem, the usual directory traversal, copying, reading, deleting files work in DOS, git bash, and explorer.

But can not read files in notepad.exe!

I have a "hello world" text file named "a.file.txt" on my filesystem drive E:, and if I try to read it in notepad, using "notepad.exe E:\a.file.txt" I get the error: "%1 is not a valid Win32 application". However, a "copy E:\a.file.txt C:\a.file.txt ; notepad.exe C:\a.file.txt" works just fine. wordpad.exe also works fine.

Since I print all IRPs coming my way, there isn't much scope for what could be wrong (I hope). In particular, running notepad.exe it only goes through a handful of queries. I have skipped CLEANUP and CLOSE, as all I do is return STATUS_SUCCESS and Information=0.

IRP_MJ_CREATE: FileObject FFFFC686327AAB20 name '\' flags 0x0
dispatcher: exit: 0x0 OK Information 0x1
query_directory_FileFullDirectoryInformation: starting Search pattern 'a.file.txt' type 3 Index 0, Restart 0, Single 1
comparing names 'a.file.txt' == 'a.file.txt'
dispatcher: exit: 0x0 OK Information 0x78
IRP_MJ_CREATE: FileObject FFFFC68632887080 name '\a.file.txt' flags 0x0
dispatcher: exit: 0x0 OK Information 0x1
fsDispatcher: enter: major 10: minor 0: IRP_MJ_QUERY_VOLUME_INFORMATION
query_volume_information: FileFsVolumeInformation
dispatcher: exit: 0x80000005 Overflow Information 0x18
fsDispatcher: enter: major 5: minor 0: IRP_MJ_QUERY_INFORMATION
query_information: FileAllInformation
   file_basic_information
   file_standard_information
   file_position_information
   * query_information: FileNameInformation
   file_name_information: remaining space 8 str.len 20 struct size 8
* file_name_information: partial name of "a"
dispatcher: exit: 0x80000005 Overflow Information 0x68
IRP_MJ_CREATE: FileObject FFFFC68631BD6BB0 name '\a.file.txt' flags 0x0
dispatcher: exit: 0x0 OK Information 0x1
fsDispatcher: enter: major 5: minor 0: IRP_MJ_QUERY_INFORMATION
* query_information: FileNetworkOpenInformation
dispatcher: exit: 0x0 OK Information 0x38
fsDispatcher: enter: major 18: minor 0: IRP_MJ_CLEANUP
dispatcher: exit: 0x0 OK Information 0x0
fsDispatcher: enter: major 2: minor 0: IRP_MJ_CLOSE
dispatcher: exit: 0x0 OK Information 0x0
IRP_MJ_CREATE: FileObject FFFFC68631AD39D0 name '\' flags 0x0
dispatcher: exit: 0x0 OK Information 0x1
fsDispatcher: enter: major 5: minor 0: IRP_MJ_QUERY_INFORMATION
query_information: FileNameInformation
* file_name_information: name of \
dispatcher: exit: 0x0 OK Information 0xa
fsDispatcher: enter: major 10: minor 0: IRP_MJ_QUERY_VOLUME_INFORMATION
* query_volume_information: FileFsVolumeInformation
dispatcher: exit: 0x0 OK Information 0x20
fsDispatcher: enter: major 10: minor 0: IRP_MJ_QUERY_VOLUME_INFORMATION
query_volume_information: FileFsAttributeInformation
dispatcher: exit: 0x0 OK Information 0x18
fsDispatcher: enter: major 5: minor 0: IRP_MJ_QUERY_INFORMATION fsDeviceObject
query_information: FileBasicInformation
dispatcher: exit: 0x0 OK Information 0x28

Now, I believe I am correct in returning OVERFLOW in FileFsVolumeInformation and FileAllInformation, but I still fill in all the information, just truncating the VolumeName and Filename respectively.

As for what I return, I set Attributes to either FILE_ATTRIBUTE_DIRECTORY or FILE_ATTRIBUTE_NORMAL - nothing else. In FileBasic/FileAll and DirectoryListing. When this was wrong, bash would show everything as character device, so I know it is now set, as they are regular files or directories.

Possibly related is that if I copy notepad.exe to E: I can not run it "This app can't run on this PC" or in bash "Exec format error". So it is as if I was returning the wrong filetype. But it is only 3 functions it calls, and only FileAttributes is related (and FileStandard's Directory boolean).

I have re-written the OVERFLOW logic a few times now, but I believe it is correct to set the fileNameLength to the total length needed to fit the name even if it didn't all fit, but Information field is the total size returned (never larger than input size) ?

What sort of sacrifice does notepad.exe need to work!?

So, notepad.exe is noteworthy in that it uses memory mapped files to read files, and mmap files were not working in my filesystem. I used TestFile from zezula.net to notice that it was not working. To enable mmap to work, I had to assign FileObject->SectionObjectPointer = &vp->SectionObjectPointers in IRP_MJ_CREATE. Most cases this is stored in your Fcb (as in FileObject->fscontext) and is just defined as SECTION_OBJECT_POINTERS SectionObjectPointers;.

Some care is also needed to flush the mmap pages in IRP_MJ_CLEANUP as per fastfat sample sources.

Also related is the setup call FsRtlRegisterFileSystemFilterCallbacks() where you want to define at least PreAcquireForSectionSynchronization callback to handle locks for sections. Also in fastfat sources.