While working on data assimilation stuff with WRF and it’s variational companion WRFDA, I have been thinking a lot about different types of errors. At it’s simplest, data assimilation is embodied in seeking a vector which minimizes a cost function that represents a linear combination of the model fit errors and the observational fit errors. The weights used in the linear combination are taken from the inverse of an “error covariance” matrix, which for the model is called B, and for the observations is typically denoted R. The assumption here is that error between the model/observations and the truth is a random variable, and that its distribution is multivariate normal with mean equal to the model/observations and covariance given by B or R, respectively.
We can get a pretty good estimate of the entries of R, since we can often do observational experiments with a particular instrument in a setting where we “know the answer”, or the truth. That these errors would be random in nature is not too troubling, if we take the example of a thermometer, which is measuring the gross motion of a bunch of molecules, and seemingly would suffer from this type of random noise due to fluctuations on the molecular scale. A similar analogy can be made for radar measurements, replacing the molecules with scatterers. If you do this experiment over and over, you can calculate the covariance of the measurement error for a large sample, and approximate R reasonably well. So this formulation for the observational error component of the problem doesn’t bother me too much.
Model error seems to me to be a philosophically different creature. First of all, the error in a model prediction depends on a lot of different factors, including the errors present in the fields you used to initialize your forecast and errors in model parameters, but also in processes that aren’t even represented in your model’s equations, or in processes that are represented incorrectly. For a particular forecast error, it’s not clear how each of these sources contributes to that error.
To better understand these ideas, I’m reading some papers that I’d like to recommend to anyone interested in data assimilation or inverse problems. The first is “Model Error in Weather Forecasting” in the journal Nonlinear Processes in Geophysics, by Orrell et al, from 2001. They address these concepts with some rigor (thankfully!). I’ll write another post soon with my synopsis and call for comments.
Sean M. R. Crowell 
Sean, I've done a bit of work in this area and would be thrilled to hear what you think, now that you're in the guts of it yourself. You were at Berkeley in 2008 I think, so maybe you've seen some of it. MWR 2007 is probably the best one. Model error has become the biggest problem area, now that data assimilation has been such a focus for so long. You won't be surprised to hear this, but I'm trying to figure out how to use tweets to reduce weather model forecast errors…
First of all, thanks for reading the scribbles that I write. Beyond that, I find myself a little perplexed by the theoretical underpinnings that go into the formulation. Granted, I don't have anything better, but I'm not certain that the answer with a 3DVar cost function is at all meaningful. In that case it almost seems as though the model error covariance is really only important as a magnitude adjustment to the obs errors (which are typically assumed diagonal anyway). I'm not super well informed yet, but I plan to focus on this more as time goes on.