Covering SPL Behaviour with Sampled Configurations: An Initial Assessment

Structural approaches to Software Product Lines (SPL) testing (such as pairwise testing) have gained momentum as they are able to scale to larger SPLs described as feature diagrams (FD). However, these methods are agnostic with respect to behaviour: the sampled configurations have thus no reason to satisfy any given behavioural criterion. In this paper, we investigate the behavioural coverage of two structural testing criteria: pairwise and similarity. To do so, we modelled four SPLs in terms of feature diagrams and associated featured transitions systems (FTSs). We then computed state, action and transition coverage for a set of generated configurations. Preliminary results indicate that for relatively small variability models with few cross-tree constraints, structural coverage-driven tools tend to cover large parts of behaviour with less than 8 configurations. Though structural coverage cannot be used directly as a replacement for behavioural driven SPL test generation, opportunities to mix structural and behavioural coverage for efficient and effective SPL testing do exist.