I wrote a short post yesterday about the notion of error statistics, and recommended a paper by Orrell, Smith, Barkmeijer and Palmer from 2001 in the journal Nonlinear Processes in Geophysics called “Model Error in Weather Forecasting.” I had read this paper in the past, though on a somewhat superficial level, and decided to read it again more carefully. As motivation, I decided to write a blog post which summarizes the main ideas of the paper.
First, the authors point out the fact that the place that a model’s prediction (x(t)) lives, meaning the variables that it predicts or diagnoses, is often a quite smaller space than the true atmosphere (X(t)) (or whatever you’re modeling). This means that when we talk about model error, we have to first project the true atmosphere into the simpler model atmosphere. They call this projection the “target orbit” (tilde{x}(t)). The term “orbit” is very commonly used in mathematics with different meanings depending on the context. I conceptualize an orbit as a collection of variables (or a vector) evolving in time. The big point to be made here is that it is likely that no initial condition we provide the model is likely to yield precisely the target orbit when we move forward in time, because the model’s dynamics are deficient. That is, the “trajectories” or orbits of the model are different than the orbit of the real thing. That’s the essence of model error. Were we to start with a perfect initial condition, the model would still diverge from the target orbit after a while. The amount of time that the model orbit stays close to the target orbit is called the “shadowing time”. In general, the best we can hope for is to maximize the shadowing time.
The major ideas in the paper are as follows:
(1) They define drift as the time integrated departure of the rate of change of the target orbit from the model tendency:
$$d(t) = int_0^tleft(G(tilde{x}(tau))-frac{dtilde{x}}{dtau}right)dtau $$
(note that this requires knowledge of the target orbit, which is nontrivial)
(2) By assuming the drift and initial condition errors are uncorrelated, they derive a linear model for error growth, and use it to derive a “shadow-drift law”, which gives a bound on the amount of time you expect a model trajectory to shadow within a given radius.
(3) They test these ideas on a series of models that increase in complexity, and find pretty good agreement with their law in all cases. There are interesting wrinkles in more realistic models, where the concept of “spinup” is important. This is the process of the model being initialized with an unbalanced initial condition, and the model having to bring itself back into balance, which leads to largish errors initially.
(4) They test out a theory that they can put both initial condition and drift errors together in a simple way, and for the complex weather model, this hypothesis is borne out by the results, meaning that the combination they come up with closely estimates the actual forecast errors as calculated from an assumed “truth”. This is a big deal, because people have always had trouble sussing out the interactions between model error and initial condition error.
Any research paper has to be read multiple times to make sure you understand it well, but I’m still missing how they come up with the target orbit. Once that’s accomplished, you can calculate the drift pretty easily, and hence get a good sense of the short range model error, assuming you also have an estimate of the initial condition error. Of course, I’m still puzzling out how I would calculate a covariance matrix using this approach, since it all is done in a nonstatistical way, which is to say, for a single case.
I would love to hear comments below. I’m sure I got some details wrong (and I know I left most of the math out), so feel free to make me clear things up below).
Based on Chris Danforth’s comment from yesterday’s post, I’m going to read his paper from 2007 in Monthly Weather Review with his advisor Eugenia Kalnay and another colleague Takemasa Miyoshi, called “Estimating and Correcting Global Weather Model Error”. Hopefully that will yield a similar result to today’s post. It’s nice to do this while my codes are running. 🙂
Sean M. R. Crowell 