I/O error while writing to HttpListenerResponse

While writing output to HttpListenerResponse, server throws I/O operation aborted exception. It occurs only when delay between request and response (i.e. time taken by DoMoreWork()) is more than 4-5 mins. There's some timeout happening but I am not able to see any log related to it.

I have set various timeout in TimeOutManager, tried turning on/off KeepAlive on both client and server side, checked logs in System32/LogFiles/Httperr but not finding anything relevant about what could be causing this error.

OS is Windows server 2016, .Net framework is 4.7.

Exception:

System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)

Windows service Code:

using (HttpListener listener = new HttpListener())
{
    listener.Prefixes.Add(/*port*/);
    listener.TimeoutManager.IdleConnection = TimeSpan.FromMinutes(15);
    listener.Start();

    while (true)
    {
        HttpListenerContext context = listener.GetContext();
        ThreadPool.QueueUserWorkItem(new WaitCallback(DoWork), context);
        Thread.Sleep(100);
    }
    listener.Stop();
}

public static void DoWork(object threadContext)
{
    HttpListenerContext context = threadContext as HttpListenerContext;
    byte[] data = DoMoreWork();
    context.Response.ContentLength64 = data .Length;

    //Does not help
    //context.Response.SendChunked = false;
    //context.Response.ContentType = System.Net.Mime.MediaTypeNames.Application.Octet;
    context.Response.OutputStream.Write(data, 0, data.Length); //THROWS
    context.Response.StatusCode = 200;
}