Empirical Assessment of Generating Adversarial Configurations for Software Product Lines

Software product line (SPL) engineering allows the derivation ofproducts tailored to stakeholders’ needs through the setting of a large numberof configuration options.Unfortunately, options and their interactions create a huge configurationspace which is either intractable or too costly to explore exhaustively. Insteadof covering all products, machine learning (ML) approximates the set of ac-ceptable products (e.g.,successful builds, passing tests) out of a training set (asample of configurations). However, ML techniques can make prediction errorsyielding non-acceptable products wasting time, energy and other resources.We applyadversarial machine learning techniquesto the world of SPLs andcraft new configurations faking to be acceptable configurations but that arenot and vice-versa. It allows to diagnose prediction errors and take appropriateactions. We develop two adversarial configuration generators on top of state-of-the-art attack algorithms and capable of synthesizing configurations thatare both adversarial and conform to logical constraints.We empirically assess our generators within two case studies: an industrialvideo synthesizer (MOTIV) and an industry-strength, open-source Web-appconfigurator (JHipster). For the two cases, our attacks yield (up to) a 100%misclassification rate without sacrificing the logical validity of adversarial con-figurations. This work lays the foundations of a quality assurance frameworkfor ML-based SPLs.