How to gracefully unload a child AppDomain that has threads running

I have a service that loads a child AppDomain and then starts a thread running in it. It needs an AppDomain because it dynamically generates and loads some code and I need to be able to restart it without killing the whole service.

So there is a thread running in an event loop in the child AppDomain, it gets events passed to it through a MarshalByRefObject that sticks stuff in a concurrent queue. I want to stop and unload the child AppDomain and create a new one.

I can simply call Unload on the child AppDomain, but that will abort all the threads and throw a ThrearAbortException. How can I gracefully shut it down? If I set some static flag in the child AppDomain using the MarshalByRefObject then how will the main process be able to wait until its done unloading?

I have some example code that kind of shows how its setup and how I can call Unload to kill it, how could I modify this to allow graceful unloading and never have multiple child AppDomains?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Security;
using System.Security.Permissions;
using System.Reflection;
using System.Threading;

namespace TestAppDomains
{
    /// <summary>
    /// Calls to methods magically get transfered to the appdomain it was created in because it derives from MarshalByRefObject
    /// </summary>
    class MarshalProxy : MarshalByRefObject
    {
        public AppDomain GetProxyAppDomain()
        {
            return AppDomain.CurrentDomain;
        }

        public void SayHello()
        {
            Console.WriteLine("MarshalProxy in AD: {0}", AppDomain.CurrentDomain.FriendlyName);
        }

        public void RunLoop()
        {
            try
            {
                while (true)
                {
                    Console.WriteLine("RunLoop {0} in {1}", DateTime.Now.ToLongTimeString(), AppDomain.CurrentDomain.FriendlyName);
                    Thread.Sleep(1000);
                }
            }
            catch(Exception ex)
            {
                Console.WriteLine("You killed me! {0}", ex);
                Thread.Sleep(200); //just to make sure the unload is really blocking until its done unloading
                // if the sleep is set to 2000 then you will get a CannotUnloadAppDomainException, Error while unloading appdomain. (Exception from HRESULT: 0x80131015) thrown from the .Unload call
            }
        }

        static int creationCount = 1;
        public static MarshalProxy RunInNewthreadAndAppDomain()
        {
            // Create the AppDomain and MarshalByRefObject
            var appDomainSetup = new AppDomainSetup()
            {
                ApplicationName = "Child AD",
                ShadowCopyFiles = "false",
                ApplicationBase = Environment.CurrentDirectory,
            };

            var childAppDomain = AppDomain.CreateDomain(
                "Child AD " + creationCount++,
                null,
                appDomainSetup,
                new PermissionSet(PermissionState.Unrestricted));

            var proxy = (MarshalProxy)childAppDomain.CreateInstanceAndUnwrap(
                typeof(MarshalProxy).Assembly.FullName,
                typeof(MarshalProxy).FullName,
                false,
                BindingFlags.Public | BindingFlags.Instance,
                null,
                new object[] { },
                null,
                null);

            Thread runnerThread = new Thread(proxy.RunLoop);
            runnerThread.Name = "MarshalProxy RunLoop";
            runnerThread.IsBackground = false;
            runnerThread.Start();

            return proxy;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("I am running in AD: {0}", AppDomain.CurrentDomain.FriendlyName);

            var proxy = MarshalProxy.RunInNewthreadAndAppDomain();
            proxy.SayHello();

            while (true)
            {
                Console.WriteLine("Press enter to kill and restart proxy");
                Console.WriteLine();
                Console.ReadLine();

                Console.WriteLine("Unloading");
                AppDomain.Unload(proxy.GetProxyAppDomain());
                Console.WriteLine("Done unloading");

                proxy = MarshalProxy.RunInNewthreadAndAppDomain();
            }
        }
    }
}

Try the following

runnerThread.IsBackground = true;

And, yeah, there is no graceful unloading of AppDomain if you didn't stop the threads first.