Deep variability modeling to enhance reproducibility of database performance testing

Reproducibility is a major principle underpinning the scientific method. Especially, deep database environment systems variability represents an opportunity for reproducibility of database performance testing where metrics defining performance are intricately woven from the entirety of the testing environment-including data sets, schema design, queries, hardware configurations, DBMS parameters, versions, code, etc. Nevertheless, despite the availability of data and code, numerous studies have highlighted a disconcerting trend: analyzing the same dataset using different software can yield divergent results. This variance stems from the nuanced interplay between explicit and implicit parameters within multiple layers, spanning the hardware platform, database model, deployment intricacies, and physical structures, showcasing the complexity inherent in performance model generation and its sensitivity to intricate interactions. Moreover, many authors of database papers do not provide ways to easily reproduce their experiments. In this paper, we propose a framework called ReFleT (REproductibility-aware Framework for LEveraging database Testing) that aims at modeling the manifold variations within the database testing environment, thereby fortifying the reproducibility of database performance models. This comprehensive framework, ReFleT, stands on three different components: (i) a reproduction design specific language, (ii) a model repository for persisting the database environment testing, and (iii) ReFleT capabilities—an interface custom-crafted for seamless navigation, provision, exploration, and reproduction of database performance testing. To substantiate the efficacy of our framework, we present a compelling proof-of-concept through its application in a meticulously designed case study.