Feature-family-based reliability analysis of software product lines

Context: Verification techniques such as model checking are being applied to ensure that software systems achieve desired quality levels and fulfill their functional and non-functional specification. However, applying these techniques to software product lines is a twofold challenge, given the exponential blowup of the number of products and the state-explosion problem inherent to model checking. Current product-line verification techniques leverage symbolic model checking and variability information to optimize the analysis but still face limitations that make them costly or infeasible. In particular, state-of-the-art verification techniques for product-line reliability analysis are enumerative which hinders their applicability, given the latent blowup of the configuration space. Objective: Our objectives are the following: (a) we present a method to eficiently compute the reliability of all configurations of a compositional or annotation-based software product line from its UML behavioral models, (b) we provide a tool that implements the proposed method, and (c) we report on an empirical study comparing the performance of different reliability analysis strategies for software product lines. Method: We present a novel feature-family-based analysis strategy to compute the reliability of all products of a (compositional or annotation-based) software product line. The strategy employs a divide-and-conquer approach over UML behavioral models endowed with probabilistic and variability information. The feature-based step of our strategy divides the behavioral models into smaller feature-dependent fragments that can be analyzed more eficiently. Such analysis consists of creating a probabilistic model for each behavioral fragment and analyzing such model using a parametric model checker that returns an expression denoting its reliability. Parameters in such expression represent the reliabilities of fragments on which it depends at runtime. The family-based step performs the reliability computation for all configurations at once (conquer) by evaluating reliability expressions in terms of a suitable variational data structure. This step solves the expression computed for each behavioral fragment taking into account (a) the fragment's variability information and (b) the reliability values already computed for the fragments on which it depends. The result is an Algebraic Decision Diagram (ADD) whose terminals different than zero represent the reliability value of valid (partial) configurations for the fragment. Therefore, the ADD computed for the last evaluated fragment contains the reliability values for all valid configurations of the software product line. Results: We performed an experiment to compare our feature-family-based and other four state-of-the-art evaluation strategies (product-based, family-based, feature-product-based and family-product-based). The subjects were variations of six publicly available product lines, whose configuration spaces were progressively increased. The empirical results show that our feature-family-based strategy outperforms, in terms of time and space, the other four state-of-the-art strategies. In addition, it is the only one that could be scaled to a 2²⁰-fold increase in the size of the configuration space. Conclusion: Our feature-family-based strategy leverages both feature-based and family-based strategies by taming the size of the models to be analyzed (due to the decomposition of behavioral models into fragments) and by avoiding the products enumeration inherent to some state-of-the-art analysis methods by using ADDs to represent both variability and reliability values. Journal paper: This paper was published at the Information and Software Technology Journal. It is available at https://doi.org/10.1016/j.infsof.2017.10.001. Supplementary material: Additional material to the IST submission is available at https://splmc.github.io/scalabilityAnalysis/. This material comprises experiments data, the tool implementing the feature-family-based reliability analysis strategy and the environment for experiment replication.