A benchmark-based evaluation of search-based crash reproduction

Mozhan Soltani, Pouria Derakhshanfar, Xavier Devroey, Arie van Deursen

Résultats de recherche: Contribution à un journal/une revueArticle

Résumé

Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43% of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.

langue originaleAnglais
journalEmpirical Software Engineering
Les DOIs
étatPublié - 1 janv. 2019
Modification externeOui

Citer ceci

Soltani, Mozhan ; Derakhshanfar, Pouria ; Devroey, Xavier ; van Deursen, Arie. / A benchmark-based evaluation of search-based crash reproduction. Dans: Empirical Software Engineering . 2019.
@article{2a2035e3bf124cfaace3ae0e0890a5b7,
title = "A benchmark-based evaluation of search-based crash reproduction",
abstract = "Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43{\%} of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.",
keywords = "Crash reproduction, Empirical evaluation, Search-based software testing",
author = "Mozhan Soltani and Pouria Derakhshanfar and Xavier Devroey and {van Deursen}, Arie",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s10664-019-09762-1",
language = "English",
journal = "Empirical Software Engineering",
issn = "1382-3256",
publisher = "Springer",

}

A benchmark-based evaluation of search-based crash reproduction. / Soltani, Mozhan; Derakhshanfar, Pouria; Devroey, Xavier; van Deursen, Arie.

Dans: Empirical Software Engineering , 01.01.2019.

Résultats de recherche: Contribution à un journal/une revueArticle

TY - JOUR

T1 - A benchmark-based evaluation of search-based crash reproduction

AU - Soltani, Mozhan

AU - Derakhshanfar, Pouria

AU - Devroey, Xavier

AU - van Deursen, Arie

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43% of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.

AB - Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43% of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.

KW - Crash reproduction

KW - Empirical evaluation

KW - Search-based software testing

UR - http://www.scopus.com/inward/record.url?scp=85072026706&partnerID=8YFLogxK

U2 - 10.1007/s10664-019-09762-1

DO - 10.1007/s10664-019-09762-1

M3 - Article

JO - Empirical Software Engineering

JF - Empirical Software Engineering

SN - 1382-3256

ER -