BOOST Unit Test stack overflow

I am now using Boost Unit Test to perform unit test for my project. Every time I run the unit test, I got a memory stack problem. I debug into the source code of BOOST library, and I find that the problem comes from invoking the following codes in unit_test_suite.hpp file:

void
traverse_test_tree( test_unit_id id, test_tree_visitor& V )
{
  global_i = global_i + 1;
   std::cout<<global_i<<std::endl;
    if( ut_detail::test_id_2_unit_type( id ) == tut_case )
        traverse_test_tree( framework::get<test_case>( id ), V );
    else
        traverse_test_tree( framework::get<test_suite>( id ), V );
}

The error information I have obtained from VC10 is:

Unhandled exception at 0x779815de in  TestApplication.exe: 0xC00000FD: Stack overflow.

I was wondering what's wrong with the test program. Thanks!

EDIT Based on the suggestions I looked through my codes, and very strange things happen: if the test suite is defined in same program with main(), it works; however, if the test suite is from a .dll, the error will occur. I list the following codes to illustrate my problem:

boost::unit_test::test_suite* main_global_test_suite;
     void Hellotestdll()
        {
            int i= 1;
            int j= 2;
            BOOST_CHECK(i == j);


        }
        boost::unit_test::test_suite* get_abc_test_suite()
        {
            test_suite* ts = BOOST_TEST_SUITE( "unit_geometric" );
            ts->add( BOOST_TEST_CASE( &Hellotestdll ) );
            return ts;
        }

int main( int argc, char* argv[] )
{
    try 
    {
        /**
        * Step 1. obtain options
        */
        char* optionLine[1024];
        int len;
        len = obtain_options(optionLine, argc, argv);
        /**
        * Step 2. perform unit test based on user's options
        */
        int test_status=0;  
        main_global_test_suite = get_abc_test_suite();  
        test_status = unit_test_main(run_global_test_suite, len, optionLine);
        return test_status;

    }
    catch(std::exception& e)
    {
        std::cout << e.what() << std::endl;
        return 1;
    }   
    catch (const std::string& s) 
    {
        std::cout << s << std::endl;
        return 1;
    }
    catch (...)
    {
        return 1;
    }


} 

The above codes work very well. But if the test suite is from a .dll, for example:

// dll_header.h
namespace abc
{

    ABC_EXPORT boost::unit_test::test_suite* get_geometric_test_suite();
}
// dll_header.cpp
namespace abc
{

    using namespace boost;
    using namespace boost::unit_test;

     void Hellotestdllabc()
       {
        int i= 1;
        int j= 2;
               BOOST_CHECK(i == j);
     }

    boost::unit_test::test_suite* get_abc_test_suite()
    {
        test_suite* ts = BOOST_TEST_SUITE( "unit_abc" );

        ts->add( BOOST_TEST_CASE( &Hellotestdllabc ) );


        return ts;
    }

}

Then if I invoke this test suite, with the following codes:

 int main( int argc, char* argv[] )
    {
         ............
            /**
            * Step 2. perform unit test based on user's options
            */
            int test_status=0;  
            main_global_test_suite = abc::get_abc_test_suite();  
            test_status = unit_test_main(run_global_test_suite, len, optionLine);
            return test_status;

        }

The annoying stack overflow error will happen.

Summery of the Problems

  (1) boost dll with MDd (Succeed) 

If I link the boost unit test library (with the definition -DBOOST_ALL_DYN_LINK -DBOOST_TEST_NO_MAIN -DBOOST_TEST_DYN_LINK -DBOOST_ALL_NO_LIB)and the running executable program with the same dynamic run-time library (Multi-thread Debug Dll (MDd)), it will work.

(2) boost dll with MTd (Failed)

If the boost unit test library (with the definition -DBOOST_ALL_DYN_LINK -DBOOST_TEST_NO_MAIN -DBOOST_TEST_DYN_LINK -DBOOST_ALL_NO_LIB)and the running executable program are compiled and link with the same static run-time libray (Multi-thred Debu (MTd)), I will have a crash, but the crash is different from the one I reported above:

(3) boost static lib with MDd (Failed)

If the boost is built as a static library (with the definition of -DBOOST_TEST_NO_MAIN -DBOOST_ALL_NO_LIB), and both the boost library and the executable program are built with the same dynamic run-time library (MDd). The following crash will happen:

(4) boost static lib with MTd (Failed)

If the boost is built as a static library (with the definition of -DBOOST_TEST_NO_MAIN -DBOOST_ALL_NO_LIB), and both the boost library and the executable program are built with the same static run-time library (MTd). The following crash will happen:

   ABC_EXPORT boost::unit_test::test_suite* get_geometric_test_suite();

The point of unit tests is to find problems in code early. That worked, it is just that you found the problem very early. Too early to even allow the unit test to run properly.

Functions in a DLL that returns pointers to C++ objects are a problem in general. It will only come to a good end when the layout of the C++ object exactly matches the assumptions made by the compiler when it compiled both your DLL and your EXE. And that the object lives on a heap that both modules have access to, required since the DLL creates the object and your EXE needs to delete it.

To allow the object to be properly deleted, both the DLL and the EXE must share the same CRT version. You will get into trouble when you build your program with /MT, asking for the static version of the CRT. The relevant compiler setting is C/C++, Code Generation, Runtime library setting. Your Debug configuration must use /MDd, your Release configuration must use /MD. For both the EXE and the DLL project, as well as the Boost library when it was compiled. If it is /MTd and /MT then the DLL will have its own copy of the CRT linked into it and will use its own heap to allocate from. The EXE cannot properly delete the object since it uses another heap. Doing so anyway will produce undefined behavior. Anything can happen, you tend to only get lucky when you run your program on a version of Windows that's newer than XP. Vista and up will use a debug heap when you run your unit test with a debugger attached, it invokes a breakpoint when it notices that the pointer passed to ::operator delete is invalid. Be sure to allow the linker to automatically find the correct Boost .lib to link to, don't force it yourself.

Object layout is more likely your problem, unfortunately much harder to diagnose. You stay out of trouble by building your EXE and DLL with the exact same compiler settings. With the additional requirement that they must match the settings that were used to build the Boost libraries. Which certainly is the difficult part, that requires a time machine. Particularly the _HAS_ITERATOR_DEBUGGING macro is a troublemaker, basic STL classes like std::vector will have a different layout that depends on the value of that macro.

I realize this is very vague, but there isn't enough info in the question to truly diagnose this issue. A very basic check you can do is putting the returned boost::unit_test::test_suite pointer in a watch expression. If you suddenly see the members of that object change when you step into Boost code then you know you have an object layout problem. What happens next is highly unpredictable, a stack overflow is certainly possible. Yet another diagnostic is to use the Debug + Windows + Registers window. Ensure that the ESP register value is stable when you step over functions.