Towards quality assurance of software product lines with adversarial configurations

Paul Temple; Mathieu Acher; Gilles Perrouin; Battista Biggio; Jean Marc Jézéquel; Fabio Roli

doi:10.1145/3336294.3336309

Towards quality assurance of software product lines with adversarial configurations

Paul Temple, Mathieu Acher, Gilles Perrouin, Battista Biggio, Jean Marc Jézéquel, Fabio Roli

Research output: Contribution in Book/Catalog/Report/Conference proceeding › Conference contribution

162 Downloads (Pure)

Abstract

Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.

Original language	English
Title of host publication	SPLC 2019 - 23rd International Systems and Software Product Line Conference
Editors	Thorsten Berger, Philippe Collet, Laurence Duchien, Thomas Fogdal, Patrick Heymans, Timo Kehrer, Jabier Martinez, Raul Mazo, Leticia Montalvillo, Camille Salinesi, Xhevahire Ternava, Thomas Thum, Tewfik Ziadi
Publisher	ACM Press
ISBN (Electronic)	9781450371384
DOIs	https://doi.org/10.1145/3336294.3336309
Publication status	Published - 9 Sept 2019
Event	23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019 - Paris, France Duration: 9 Sept 2019 → 13 Sept 2019

Publication series

Name	ACM International Conference Proceeding Series
Volume	A

Conference

Conference	23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019
Country/Territory	France
City	Paris
Period	9/09/19 → 13/09/19

Funding

∗Gilles Perrouin is an FNRS Research Associate. This research was partly supported by EOS Verilearn project grant no. O05518F-RG03. This research was also funded by the ANR-17-CE25-0010-01 VaryVary project.

Keywords

Machine learning
Quality assurance
Software product line
Software testing
Software variability

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1145/3336294.3336309

Adversarial_Constraints_for_Variability_Models_SPLC2019Accepted author manuscript, 22 MB

Cite this

Temple, P., Acher, M., Perrouin, G., Biggio, B., Jézéquel, J. M., & Roli, F. (2019). Towards quality assurance of software product lines with adversarial configurations. In T. Berger, P. Collet, L. Duchien, T. Fogdal, P. Heymans, T. Kehrer, J. Martinez, R. Mazo, L. Montalvillo, C. Salinesi, X. Ternava, T. Thum, & T. Ziadi (Eds.), SPLC 2019 - 23rd International Systems and Software Product Line Conference (ACM International Conference Proceeding Series; Vol. A). ACM Press. https://doi.org/10.1145/3336294.3336309

Temple, Paul ; Acher, Mathieu ; Perrouin, Gilles et al. / Towards quality assurance of software product lines with adversarial configurations. SPLC 2019 - 23rd International Systems and Software Product Line Conference. editor / Thorsten Berger ; Philippe Collet ; Laurence Duchien ; Thomas Fogdal ; Patrick Heymans ; Timo Kehrer ; Jabier Martinez ; Raul Mazo ; Leticia Montalvillo ; Camille Salinesi ; Xhevahire Ternava ; Thomas Thum ; Tewfik Ziadi. ACM Press, 2019. (ACM International Conference Proceeding Series).

@inproceedings{1b4dc6fad1ab4dbc8517466e2356a51c,

title = "Towards quality assurance of software product lines with adversarial configurations",

abstract = "Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100\% misclassification rate and a drop in accuracy of 5\%. We discuss the implications these results have on SPL quality assurance.",

keywords = "Machine learning, Quality assurance, Software product line, Software testing, Software variability",

author = "Paul Temple and Mathieu Acher and Gilles Perrouin and Battista Biggio and J{\'e}z{\'e}quel, \{Jean Marc\} and Fabio Roli",

note = "Funding Information: This work has been partially funded by the Ministry of Science and Technology of Spain through ECLIPSE (RTI2018-094283-B-C33), the Junta de Andaluc?a via the PIRAMIDE and METAMORFOSIS projects, the European Regional Development Fund (ERDF/FEDER), and the MINECO Juan de la Cierva postdoctoral program. The authors would like to thank the C?tedra de Telef?nica {"}Inteligencia en la Red{"} of the Universidad de Sevilla for its support. Funding Information: ∗Gilles Perrouin is an FNRS Research Associate. This research was partly supported by EOS Verilearn project grant no. O05518F-RG03. This research was also funded by the ANR-17-CE25-0010-01 VaryVary project. Publisher Copyright: {\textcopyright} 2019 Association for Computing Machinery. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.; 23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019 ; Conference date: 09-09-2019 Through 13-09-2019",

year = "2019",

month = sep,

day = "9",

doi = "10.1145/3336294.3336309",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "ACM Press",

editor = "Thorsten Berger and Philippe Collet and Laurence Duchien and Thomas Fogdal and Patrick Heymans and Timo Kehrer and Jabier Martinez and Raul Mazo and Leticia Montalvillo and Camille Salinesi and Xhevahire Ternava and Thomas Thum and Tewfik Ziadi",

booktitle = "SPLC 2019 - 23rd International Systems and Software Product Line Conference",

address = "United States",

}

Temple, P , Acher, M , Perrouin, G, Biggio, B, Jézéquel, JM & Roli, F 2019, Towards quality assurance of software product lines with adversarial configurations. in T Berger, P Collet, L Duchien, T Fogdal, P Heymans, T Kehrer, J Martinez, R Mazo, L Montalvillo, C Salinesi, X Ternava, T Thum & T Ziadi (eds), SPLC 2019 - 23rd International Systems and Software Product Line Conference. ACM International Conference Proceeding Series, vol. A, ACM Press, 23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019, Paris, France, 9/09/19. https://doi.org/10.1145/3336294.3336309

Towards quality assurance of software product lines with adversarial configurations. / Temple, Paul ; Acher, Mathieu ; Perrouin, Gilles et al.
SPLC 2019 - 23rd International Systems and Software Product Line Conference. ed. / Thorsten Berger; Philippe Collet; Laurence Duchien; Thomas Fogdal; Patrick Heymans; Timo Kehrer; Jabier Martinez; Raul Mazo; Leticia Montalvillo; Camille Salinesi; Xhevahire Ternava; Thomas Thum; Tewfik Ziadi. ACM Press, 2019. (ACM International Conference Proceeding Series; Vol. A).

Research output: Contribution in Book/Catalog/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Towards quality assurance of software product lines with adversarial configurations

AU - Temple, Paul

AU - Acher, Mathieu

AU - Perrouin, Gilles

AU - Biggio, Battista

AU - Jézéquel, Jean Marc

AU - Roli, Fabio

N1 - Funding Information: This work has been partially funded by the Ministry of Science and Technology of Spain through ECLIPSE (RTI2018-094283-B-C33), the Junta de Andaluc?a via the PIRAMIDE and METAMORFOSIS projects, the European Regional Development Fund (ERDF/FEDER), and the MINECO Juan de la Cierva postdoctoral program. The authors would like to thank the C?tedra de Telef?nica "Inteligencia en la Red" of the Universidad de Sevilla for its support. Funding Information: ∗Gilles Perrouin is an FNRS Research Associate. This research was partly supported by EOS Verilearn project grant no. O05518F-RG03. This research was also funded by the ANR-17-CE25-0010-01 VaryVary project. Publisher Copyright: © 2019 Association for Computing Machinery. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.

PY - 2019/9/9

Y1 - 2019/9/9

N2 - Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.

AB - Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.

KW - Machine learning

KW - Quality assurance

KW - Software product line

KW - Software testing

KW - Software variability

UR - https://www.scopus.com/pages/publications/85072892794

U2 - 10.1145/3336294.3336309

DO - 10.1145/3336294.3336309

M3 - Conference contribution

AN - SCOPUS:85072892794

T3 - ACM International Conference Proceeding Series

BT - SPLC 2019 - 23rd International Systems and Software Product Line Conference

A2 - Berger, Thorsten

A2 - Collet, Philippe

A2 - Duchien, Laurence

A2 - Fogdal, Thomas

A2 - Heymans, Patrick

A2 - Kehrer, Timo

A2 - Martinez, Jabier

A2 - Mazo, Raul

A2 - Montalvillo, Leticia

A2 - Salinesi, Camille

A2 - Ternava, Xhevahire

A2 - Thum, Thomas

A2 - Ziadi, Tewfik

PB - ACM Press

T2 - 23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019

Y2 - 9 September 2019 through 13 September 2019

ER -

Temple P , Acher M , Perrouin G, Biggio B, Jézéquel JM, Roli F. Towards quality assurance of software product lines with adversarial configurations. In Berger T, Collet P, Duchien L, Fogdal T, Heymans P, Kehrer T, Martinez J, Mazo R, Montalvillo L, Salinesi C, Ternava X, Thum T, Ziadi T, editors, SPLC 2019 - 23rd International Systems and Software Product Line Conference. ACM Press. 2019. (ACM International Conference Proceeding Series). doi: 10.1145/3336294.3336309

Towards quality assurance of software product lines with adversarial configurations

Abstract

Publication series

Conference

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this