Interpretability and variability of metamodel validation statistics in engineering system design optimization: a practical study

Abstract

Prediction accuracy of a metamodel of an engineering system in comparison to the simulation model it approximates is one fundamental criterion that is used in metamodel validation. Many statistics are used to quantify prediction accuracy of metamodels in deterministic simulations. The most frequently used ones include the root-mean-square error (RMSE) and the R-square metric derived from it, and to a lesser degree the average absolute error (AAE) and its derivates such as the relative average absolute error (RAAE). In this paper, we compare two aspects of these statistics: interpretability of results returned by these statistics and their sample-to-sample variations, putting more emphasis on the latter. We use the difference-mode to common-mode ratio (DMCMR) as a measure of sample-to-sample variations for these statistics. Preliminary results are obtained and discussed via a number of analytic and electronic engineering examples.

Keywords

• simulation
• modeling
• metamodel validation