meltin.net

CTDB, the Clustered Trivial Database, is used to support clustered versions of Samba, NFS, HTTP and other protocols. It handles node monitoring, network failover and synchronisation of state between nodes. As such, there are a lot of moving parts and automated testing is non-trivial.

We present details of some testing techniques being used by CTDB. Given that CTDB normally runs as root we have tried to find testing methods that allow a developer to run "make test" as a regular user in the development tree and do a useful amount of testing. One unit testing technique involves scripted stub replacements to replace various system commands, allowing event scripts to be tested in a sandbox. We were going to build a shared library containing most of the CTDB code and replace some functions with stubs using LD_PRELOAD, allowing a useful subset of CTDB's functions to be unit tested. Instead we decided to build CCAN-style test programs that "just" #include all the relevant source code - this makes it easy to test static functions. For system testing, we also discuss our use of KVM-based virtual clusters and our trivial configuration tool to make it easier to setup such clusters. The techniques probably aren't original or mindblowing but should still be of interest to many developers. :-)