in-sample

The continuous feedback from making forecasts on new unseen out-of-sample data is largely denied the climate modeller, who is constrained by the nature of the problem to forever violate one of the first maxims of undergraduate statistics: never restrict your analysis to in-sample statistics.

Unthreaded #5 « Climate Audit 2007

By construction, climate modelling is an in-sample science.

Unthreaded #5 « Climate Audit 2007

But if differences in the subtle details of different models that have similar plausible in-sample performance are shown to yield significantly different forecast distributions, then no coherent picture emerges from the overall ensemble.

Unthreaded #5 « Climate Audit 2007

The EU approach appears to be dependent on those adjustment factors developed in-sample.

YTD Hurricane Activity « Climate Audit 2007

In any event, we can still use these differences in the forecast distributions to gain physical insight, and improve each model individually using the in-sample data yet again.

Unthreaded #5 « Climate Audit 2007

But by studying the in-sample behaviour of ensembles under a variety of models, we can tune each model until an ensemble of initial conditions under each and every individual model can at least bound the in-sample observations, say from 1950 to 2000.

Unthreaded #5 « Climate Audit 2007

I know stock pickers who know way more than I do about stocks and the underlying corporations, but if they tell me that based on in-sample data that they can demonstrate this unfailingly successful stock selection filter i.e. a process very susceptible to data snooping and model over fitting, I do not have to be their equal in stock knowledge to know enough to be very wary of their advice.

The Dendroclimatologists are Angry « Climate Audit 2007

To my mind, this study appears very analogous to an investment strategy that has been derived from data mining and the over fit nature of the in-sample results becoming apparent only after sufficient years of seeing out-of-sample results.

Groveman and Landsberg « Climate Audit 2006

The claims and methods for the HS certainly show much evidence of data mining, but since in their case the in-sample results show little or no correlation, out-of-sample results will be of little evaluative worth.

Groveman and Landsberg « Climate Audit 2006

But avoidance of explicit specification search will not cure our discomfort with sloppy in-sample ‘œtests’ nor will it insulate us from dishonest or deluded results.

Greene: I am not, nor have I ever been a member of a data-mining discipline « Climate Audit 2006