meltin.net

In this talk we will describe a fully automated test environment for clustered systems. The motivation for building this test environment was to provide high quality automatic testing of IBM's 'SoFS' (Scale-out File Services) clustered NAS product.

At the core of this test environment is autocluster, a system that can generate a pristine virtual test cluster from a base image in less than a minute. A base image built from a distro ISO image can be automatically built in about 15 minutes from a simple configuration file, so you can be using autocluster within 30 minutes of downloading it. The autocluster system uses templates to automatically configure such complex elements as a clustered filesystem, clustered Samba, multipath disk access and hierarchical storage management.

On top of autocluster we have built a full test environment that provides automated testing of NAS functionality, including scripting of Windows clients, domain controllers and automatic fail-over of cluster nodes after simulated hardware failure.

Autocluster also makes it simple to create custom environments for reproducing and debugging individual problems as they are reported. The system uses KVM via the libvirt command-line utilities. Our presentation will include demonstrations of how virtual clusters can be created using autocluster and how those clusters can be used in automated testing.