Boost Unit Test Reporting

In case it’s not obvious, I’m doing some work with boost today upgrading from 1.34.1 -> 1.42.0. I just came across this following error while running my tests

<pre>C:\projects\svn\bdk\ot51>ZenithBase_test.exe –report_level=yes
Test setup error: invalid report level yes</pre>

Thats funny because ‘yes’ used to be a valid option and I can’t see what they’ve changed it to or even what the possible options are anywhere on the internet… so I decided to put them down here based on my digging through the source:

* confirm – CONFIRMATION_REPORT,
* short – SHORT_REPORT,
* detailed – DETAILED_REPORT,
* no – NO_REPORT

BOOST_STATIC_ASSERT incomplete type not allowed

I’ve recently been trying to compile the newest boost library (1.42.0) using the Intel 11.0 compiler and came across this error in code which previously compiled using 1.34.1

1>../../deps/boost_1_42_0/boost/test/floating_point_comparison.hpp(229): error: incomplete type is not allowed
1>          BOOST_STATIC_ASSERT( !is_integral::value );
1>          ^
1>          detected during:
1>            instantiation of "boost::test_tools::predicate_result boost::test_tools::check_is_close_t::operator()(FPT1, FPT2, boost::test_tools::percent_tolerance_t, boost::test_tools::floating_point_comparison_type) const [with FPT1=int, FPT2=int, ToleranceBaseType=double]" at line 523 of "../../deps/boost_1_42_0/boost/test/test_tools.hpp"
1>            instantiation of "bool boost::test_tools::tt_detail::check_frwd(Pred, const boost::unit_test::lazy_ostream &, boost::test_tools::const_string, size_t={unsigned int}, boost::test_tools::tt_detail::tool_level, boost::test_tools::tt_detail::check_type, const Arg0 &, const char *, const Arg1 &, const char *, const Arg2 &, const char *) [with Pred=boost::test_tools::check_is_close_t, Arg0=int, Arg1=int, Arg2=boost::test_tools::percent_tolerance_t]" at line 26 of
1>                      "..\tests\Timing\VlcTimingProfile_test.cpp"

incomplete type huh, what the hell could that be about? Well it turns out that this is just a very *un*informative error which is generated when the assert evaluated to false. In my case it was caused by calling BOOST_CHECK_CLOSE(int,int,float), when I changed this to BOOST_CHECK_CLOSE(float,float,float) it compiled fine… but the error message didn’t really reflect where the error was at all.

Boost tokenize for the win

String manipulation has historically been a major shortfalling of the C++ language, or at least it was until boost came along. Something as simple as spliting a string up on a delimeter is now made so much easier thanks to the boost::tokenizer and boost string algorithms.

#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>

using namespace std;
using namespace boost;

int main(int argc, char** argv)
{
   string text = "token, test   string";

   char_separator<char> sep(", ");
   tokenizer<char_separator<char>> tokens(text, sep);
   for (tokenizer<char_separator<char> >::iterator it = tokens.begin(); it != tokens.end(); ++it)
   {
      cout << *it << "." << endl;
   }
}

And my personal favourite is to push the result of the tokenization into a vector/list/stl container.

#include 
#include 
#include 
#include 
#include 

using namespace std;
using namespace boost;

int main(int argc, char** argv)
{
   string text = "token, test   string";

   list tokenList;
   split(tokenList, text, is_any_of(", "), token_compress_on);
   (list::iterator it = tokenList.begin(); it != tokenList.end(); ++it)
   {
      cout << *it << "." << endl;
   }
}

So awesome!

Stolen from the lazy programmer and reposted here for my own reference.

More STLPort and Boost problems

As a follow up to a previous post I had about getting boost to play nicely with stlport (here), I decided that I’d like to do all of this with the Intel compiler rather than the standard VIsual Studio compiler… needless to say this created, headaches lets call them.

From the boost getting started page there is a section describing the ABI-tag that is used to describe the build options used to create a library:

• s linking statically to the C++ standard library and compiler runtime support libraries.
• g using debug versions of the standard and runtime support libraries.
• y using a special debug build of Python.
• d building a debug version of your code.
• p using the STLPort standard library rather than the default one supplied with your compiler.
• n using STLPort’s deprecated “native iostreams” feature.

however there is no helpful resource I can find to show you how to specifically build these using the boost bjam build system so I’m going to try and summarise them here.

• s => link=static – This should give static (as opposed to dynamic linking) UPDATE for 1.42 you should also use runtime-link=static
• mt => threading=multi – This will compile against the multi-threaded version of runtime libraries
• p => stdlib=stlport – This compiles against the STLport version of runtime libraries instead of the default “native”
• g => runtime-debugging=on

But for the life of me I couldn’t figure out what command/switch would get the g-option compiled into the library… now this was a problem because when I compiled my own code, the boost autolinking required a ‘mt-gd’ version of the library and I could only compile a ‘mt-d’ version. According to this page from boost, simply building against the debug variant should get you

off on off on

After spending too many hours trying to figure this out, I simply renamed the ‘mt-d’ version of my libraries to ‘mt-gd’ and the auto linking seemed to work fine

I’m also repeating here the regular command line I ended up using to compile the bloody thing…

cd "\\Program Files\\boost\\boost_1_34_1\\libs\\test\\build"
del /Q ..\\..\\..\\bin.v2\\libs\\test\\build\\intel-win-10.1\\debug\\link-static\\threading-multi\\*
bjam --toolset=intel link=static threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\test\\build\\intel-win-10.1\\debug\\link-static\\threading-multi\\libboost_unit_test_framework-iw-mt-d-1_34_1.lib c:\\bdk\\Source\\deps\\boost_1_34_1\\lib\\Intel
bjam --toolset=intel link=static threading=multi release
copy /Y ..\\..\\..\\bin.v2\\libs\\test\\build\\intel-win-10.1\\release\\link-static\\threading-multi\\libboost_unit_test_framework-iw-mt-1_34_1.lib   c:\\bdk\\Source\\deps\\boost_1_34_1\\lib\\Intel

and then because I was also using the filesystem library I had to recompile that aswell using the same form.

cd "\\Program Files\\boost\\boost_1_34_1\\libs\\filesystem\\build"
del /Q ..\\..\\..\\bin.v2\\libs\\filesystem\\build\\intel-win-10.1\\debug\\link-static\\threading-multi\\*
bjam --toolset=intel link=static threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\filesystem\\build\\intel-win-10.1\\debug\\link-static\\threading-multi\\libboost_filesystem-iw-mt-d-1_34_1.lib c:\\bdk\\Source\\deps\\boost_1_34_1\\lib\\Intel

Boost + STLport + Visual Studio 2005

Okay so here is the problem I’ve been having, I’m trying to use STLport (5.1.4), Boost (1.34.1) and Visual Studio 2005 and up to now they were not playing nice with one another. Specifically it was the boost::unit_test_framework that was causing the hassles as it has to be recompiled with STLport support. I’ve finally got it working (I think) so I’ll give you the quick run down on what I did

The short version

1. I changed the runtime library from MTd (Multi-threaded debug) to MDd (Multi-threaded debug DLL)
2. I added __STL_DEBUG to the list of preprocessor definitions for all my projects
3. I recompiled the test lib with the following command

cd "\\Program Files\\boost\\boost_1_34_1\\libs\\test\\build"
del /Q ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\stdlib-stlport\\threading-multi\\*
bjam --toolset=msvc link=static stdlib=stlport threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\stdlib-stlport\\threading-multi\\libboost_unit_test_framework-vc80-mt-gdp-1_34_1.lib ..\\..\\..\\lib
bjam --toolset=msvc link=static stdlib=stlport threading=multi build=yes release
copy /Y ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\release\\asynch-exceptions-on\\link-static\\stdlib-stlport\\threading-multi\\libboost_unit_test_framework-vc80-mt-gp-1_34_1.lib ..\\..\\..\\lib

4. and then because I was also using the filesystem library I had to recompile that aswell using the same form.

cd "\\Program Files\\boost\\boost_1_34_1\\libs\\filesystem\\build"
del /Q..\\..\\..\\bin.v2\\libs\\filesystem\\build\\msvc-8.0\\debug\\link-static\\stdlib-stlport\\threading-multi\\*
bjam --toolset=msvc link=static stdlib=stlport threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\filesystem\\build\\msvc-8.0\\debug\\link-static\\stdlib-stlport\\threading-multi\\libboost_filesystem-vc80-mt-gdp-1_34_1.lib ..\\..\\..\\lib

The long version
The host program that I am writing the plugins for provides an interface which uses a modified version of vector.h from the one that comes with visual studio (probably because they are using borland but I don’t think that they are using a stock borland vector.h either). This file comes with their dll interface and must be included with the #include “vector.h” rather than the standard #include (this will search the regular include search path first). The problem with this is that this vector.h causes problems when using boost::unit_test system and the problem was coming down to the order in which I included some files… put before the call to the interface header and it would work in boost::test but would crash from the host program, and vice versa for the alternate include order (much hair pulling to figure that one).

So my solution was to use the stlport alternative to stl – I’d used this previously for another plugin and knew that it worked just as well as the hacked version of vector.h and would hopefully also work with boost… right ? right? WRONG! well to start with anyway. Once I had stlport compiled, installed and correctly referenced on the Visual Studio search paths, I noticed some nice zero muss output coming out of the auto linking functionality

...
DemographicAllocator.cpp
STLport: Auto linking to stlportstld.5.1.lib
Generating Code...
Creating library...
BeachAssignment_logic - 0 error(s), 0 warning(s)

And the same thing happened for my boost::test project, until it came time for the linking stage at which point I was told

Unable to find library libboost_unit_test_framework-vc80-mt-sgdp-1_34_1.lib

I had used the boost consulting prebuilt binaries for windows and had up until this point been very happy, but alas my usage is apparently in the minority and they don’t even have the option to download that particular lib… so I’d have to build it myself – welcome to the world of bjam PAIN!!! Bjam is the Boost modified version of Perforce Jam build tool and I mucked about with it for a very long time until I finally got something close to the desired library to build using the following command:

C:\\Program Files\\boost\\boost_1_34_1\\libs\\test\\build>bjam --toolset=msvc-stlport runtime-link=static link=static stdlib=stlport threading=multi
WARNING: No python installation configured and autoconfiguration
         failed.  See http://www.boost.org/libs/python/doc/building.html
         for configuration instructions or pass --without-python to
         suppress this message and silently skip all Boost.Python targets
warning: Python location is not configured
warning: the Boost.Python library won't be built
Building Boost.Regex with the optional Unicode/ICU support disabled.
Please refer to the Boost.Regex documentation for more information
(don't panic: this is a strictly optional feature).
...patience...
...found 531 targets...
...updating 6 targets...
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\cpp_main.obj cpp_main.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
msvc.archive ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\libboost_prg_exec_monitor-vc-mt-sgdp-1_34_1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\exception_safety.obj exception_safety.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\interaction_based.obj interaction_based.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\logged_expectations.obj logged_expectations.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
msvc.archive ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-stlport\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\libboost_unit_test_framework-vc-mt-sgdp-1_34_1.lib
...updated 6 targets...

and then with a bit of sleuth work I got the following

C:\\Program Files\\boost\\boost_1_34_1\\libs\\test\\build>bjam --toolset=msvc runtime-link=static link=static stdlib=stlport threading=multi
WARNING: No python installation configured and autoconfiguration
         failed.  See http://www.boost.org/libs/python/doc/building.html
         for configuration instructions or pass --without-python to
         suppress this message and silently skip all Boost.Python targets
warning: Python location is not configured
warning: the Boost.Python library won't be built
Building Boost.Regex with the optional Unicode/ICU support disabled.
Please refer to the Boost.Regex documentation for more information
(don't panic: this is a strictly optional feature).
...patience...
...found 531 targets...
...updating 6 targets...
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\cpp_main.obj cpp_main.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
msvc.archive ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\libboost_prg_exec_monitor-vc80-mt-sgdp-1_34_1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\exception_safety.obj exception_safety.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\interaction_based.obj interaction_based.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
compile-c-c++ ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\logged_expectations.obj logged_expectations.cpp
STLport: Auto linking to stlportstld_x.5.1.lib
msvc.archive ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\runtime-link-static\\stdlib-stlport\\threading-multi\\libboost_unit_test_framework-vc80-mt-sgdp-1_34_1.lib
...updated 6 targets...

I say close to the desired library because even after copying it to the correct library and linking against it I was getting unresolved external errors which I had no idea of how to fix, which is when I started from scratch and found that if I used the default runtime libraries from /MTd (Multi-threaded debug) to /MDd (Multi-threaded debug DLL) that I was now able to get through this problem… after I’d recompiled those two libraries again using the following command to use shared library linking against the common runtime

cd "\\Program Files\\boost\\boost_1_34_1\\libs\\test\\build"
del /Q ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\stdlib-stlport\\threading-multi\\*
bjam --toolset=msvc link=static stdlib=stlport threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\test\\build\\msvc-8.0\\debug\\asynch-exceptions-on\\link-static\\stdlib-stlport\\threading-multi\\libboost_unit_test_framework-vc80-mt-gdp-1_34_1.lib ..\\..\\..\\lib
cd "\\Program Files\\boost\\boost_1_34_1\\libs\\filesystem\\build"
del /Q..\\..\\..\\bin.v2\\libs\\filesystem\\build\\msvc-8.0\\debug\\link-static\\stdlib-stlport\\threading-multi\\*
bjam --toolset=msvc link=static stdlib=stlport threading=multi
copy /Y ..\\..\\..\\bin.v2\\libs\\filesystem\\build\\msvc-8.0\\debug\\link-static\\stdlib-stlport\\threading-multi\\libboost_filesystem-vc80-mt-gdp-1_34_1.lib ..\\..\\..\\lib

Now I can compile correctly but for some reason there seems to be some memory leaks that weren’t there before …? WTF?

Memory Leak Hunting Season (be wery wery quiet)

The leaks would appear to be caused by the intelligent node allocation implemented in stlport, [[http://stlport.sourceforge.net/FAQ.shtml#leaks the FAQ]] notes this and calls these “pseudo” memory leaks as the memory is released, just not when the visual studio memory leak finding logic would expect it to be released. You can force STLport to use a more strict memory release mechanism by editing the file “${STL_ROOT}/stlport/stl/config/host.h” and uncommenting the following lines

/*
 * Uncomment _STLP_LEAKS_PEDANTIC to force deallocation of ALL allocated
 * memory chunks. Normally not required. But if you worry about quazi-leaks
 * (may be reported by some leaks detection tools), use
 * _STLP_LEAKS_PEDANTIC. It should be used with _STLP_USE_NEWALLOC or
 * _STLP_USE_MALLOC (see below), the default node_alloc allocator also clean
 * its internal memory pool but only if STLport is used as a dynamic library
 * under Win32 (using MSVC like compilers).
 */
#define _STLP_LEAKS_PEDANTIC 1

/*
 * Uncomment _STLP_USE_NEWALLOC to force allocator to use plain "new"
 * instead of STLport optimized node allocator engine.
 */
#define _STLP_USE_NEWALLOC 1

then recompile, install, recompile the boost libraries and recompile your application…. this resulted in the following change in memory leakage:

C:\\Program Files\\STLport-5.1.4\\bin>"c:\\BDK\\Source\\BeachAssignment\\vc7\\Debug\\BeachAssignment_test.exe"
Running 1 test case...

*** No errors detected
Detected memory leaks!
Dumping objects ->
{182} normal block at 0x00379DA0, 2616 bytes long.
 Data:  C8 A1 37 00 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3
{168} normal block at 0x00378A90, 3968 bytes long.
 Data:  B0 91 37 00 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3
{167} normal block at 0x00378790, 720 bytes long.
 Data:  C0 88 37 00 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3
{164} normal block at 0x00378180, 1280 bytes long.
 Data:  A0 83 37 00 6C 61 74 69 6F 6E 53 74 61 74 69 73
Object dump complete.

C:\\Program Files\\STLport-5.1.4\\bin>"c:\\BDK\\Source\\BeachAssignment\\vc7\\Debug\\BeachAssignment_test.exe"
Running 1 test case...

*** No errors detected
Detected memory leaks!
Dumping objects ->
{164} normal block at 0x00378180, 1280 bytes long.
 Data:  A0 83 37 00 6C 61 74 69 6F 6E 53 74 61 74 69 73
Object dump complete.

Hmmm, there is still one block of memory that hasn’t been released but I can not yet figure out where or why it is showing up… that is a job for tomorrow!