In the mid-twentieth century, George Box and others argued convincingly that model misspecification errors or "bias" should dominate our thinking about planning experiments. The reason was that in the problems they studied, bias had a far greater effect on accuracy than variance and efforts to mitigate the effects of bias generally helped with other errors, but not vice versa. Yet, fifty years later, researchers are just beginning to include bias considerations in the planning of experiments and the analysis of data. Perhaps the main complicating issue related to bias is the need to declare assumptions about the system "a priori" in the Bayesian fashion. We begin with a review of previous research about bias in experimental planning, including the definition of bias, assumptions about bias, the effect of bias, and several bias criteria that are used to obtain optimal designs and evaluate bias sensitivity, including for irregularly shaped design regions.
Using regression to analyze "on-hand" data is more common than uses after planned experiments. Yet, in both cases, available approaches to estimate the "bias susceptibility" of the fitted model are limited. To provide diagnostic information about the bias and other summative information, we propose a model diagnostic that can be used like adjusted R2 but which explicitly accounts for bias errors. Unlike the Cp statistic, our proposed diagnostic can be estimated even if the bias sources are inestimable using ordinary least squares. The proposed diagnostic has the simple interpretation of being the expected plus or minus prediction errors in the units of the response. The diagnostic, which is based on Bayes' Theorem, can be used for ordinary least squares model selection giving rise to what we call "semi-Bayesian" regression.
The key idea of the proposed diagnostic is to apply Bayesian regression to derive a picture of the bias sources for the fitted model. For this reason, we also provide a systematic analysis of the robustness of alternative Bayesian regression priors with the intent of providing generally applicable assumptions for "typical" regression applications. Two case studies involving furniture systems design are used to illustrate the proposed methods.