Exception thrown, except if run after breakpoint

I am running into some odd behavior when attempting to delete a file. Below is a quick code sample:

if (File.Exists(filePath))
{
    File.Delete(filePath);
}

The odd behavior is a System.IO.IOException is thrown when attempting to delete the file:

System.IO.IOException
HResult=0x80070020
Message=The process cannot access the file 'C:\ProgramData\Path\To\File.sqlite' because it is being used by another process.
Source=mscorlib

In our application we only access files in using statements to avoid any persistent holds. The file is a sqlite database, there are no persistent or open connections.

This seems like a pretty normal issue. The odd behavior starts when adding breakpoints to the code. For example, if we add a Console.WriteLine statement above, break on that statement, then continue:

<x> Console.WriteLine("Break");
    if (File.Exists(filePath))
    {
        File.Delete(filePath);
    }

No exception is thrown.

If we add the breakpoint like this:

    Console.WriteLine("Break");
    if (File.Exists(filePath))
    {
<x>     File.Delete(filePath);
    }

or like this:

    Console.WriteLine("Break");
<x> if (File.Exists(filePath))
    {
        File.Delete(filePath);
    }

The exception is thrown.

Other than changing the breakpoints, everything is run exactly the same. We can reproduce this behavior reliably. I've done some research into this behavior, and unable to find anything that applies. I'm curious if this behavior is documented anywhere, or if I am missing anything obvious.

Edit: Some notes I forgot to add, we can reproduce this behavior on multiple different machines. Adding a Thread.Sleep() does not change the behavior.

Any insight is greatly appreciated!

After a few hours, some Google, and many hairs pulled out, I have figured out the cause of this behavior.

From what I have found, this issue was caused by Garbage Collection not completing when we expected.

All of our SQLite statements are wrapped in using, ensuring Dispose() is called. For some reason, it does not look like Garbage Collection was being run, and some file locks remained. When we added breakpoints, it looks like there was something kicking off Garbage Collection behind the scenes.

Adding the following was enough to get things working as we expected:

GC.WaitForPendingFinalizers();