Abstract
Many approaches for testing configurable software systems start from the same assumption:it is impossible to test all configurations. This motivated the definition of variability-aware
abstractions and sampling techniques to cope with large configuration spaces. Yet, there
is no theoretical barrier that prevents the exhaustive testing of all configurations by
simply enumerating them, if the effort required to do so remains acceptable. Not only
this: we believe there is lots to be learned by systematically and exhaustively testing a
configurable system. We report on our endeavor to test all possible configurations of an
industry-strength, open source configurable software system, JHipster, a popular code
generator for web applications. We built a testing scaffold for the 26,000+ configurations
of JHipster using a cluster of 80 machines for a total of 4376 hour-machine. We find that
34.37% configurations fail and we identify the feature interactions that cause the errors.
We show that sampling testing strategies (like dissimilarity and 2-wise) (1) are more
effective to find faults than the 12 default configurations used in the JHipster continuous
integration; (2) can be too costly and exceed the available testing budget. We cross this
quantitative analysis with the qualitative assessment of JHipster's lead developers.
Key words: Case Study; Web-apps; Variability-related Analyses; JHipster; Software Product
Line; Software Product Line Testing; t-wise criteria;
| Date of Award | 26 Jun 2017 |
|---|---|
| Original language | French |
| Awarding Institution |
|
| Supervisor | Anthony Cleve (President), Patrick Heymans (Supervisor), Gilles Perrouin (Co-Supervisor) & Xavier Devroey (Co-Supervisor) |