Reality Check at SIAM UQ

I was at the SIAM Uncertainty Quantification conference on April 2 and 3rd, and after pondering my experience there for a couple of weeks, I decided there was something I wanted to talk about.  I don’t think that anyone expects applied mathematicians to only work with large models, or badly constrained problems, but there was a surprising lack of those things in the sessions I attended.  Most importantly, all of the experiments I heard about were of the “identical twin” variety.

Identical twin experiments use a model to generate a set of “observations”, add random noise to the simulated observations, and then try to estimate the correct state, or the correct model parameters, or something, using these noisy observations and the SAME model.  This type of experiment is a natural first step when you’re trying out a new method of assimilating observations, because you want to limit the sources of error you have to consider to make sure the method is actually working.

The “but” comes in here.  Identical twin experiments give an extremely optimistic prediction of how well a method will work.  The truth is that in an operational setting, the observations are coming from an unknown space (the range of the “truth model”).  Our models are much lower dimensional than the thing we’re attempting to understand, and the mismatch between the “real thing” and our models creates a huge hurdle to overcome for any method.  By doing only identical twin experiments, you never see how the new technique performs with this large extra source of error.

What other kinds of experiments are there?  The most common alternative is to use two competing models, one as the “truth”, and the other as the model to be improved using observations.  This is still an imperfect test,  but at least there is a real statistical difference in the types of errors the two models are making.  Admittedly, the difference between any two models of the phenomenon is going to be smaller than the difference between either model and the true state of the system.

The disconnect in the math community made itself known when I gave my talk.  It was more of a science talk, since there were no equations, and lots of pictures and talk about the observing systems I was investigating.  I caused a lot of confusion when I made reference to observations being “tossed out” by the data assimilation software.  The audience was genuinely concerned that the model had no right to do such a thing!  The truth is that even small observation errors could cause very unphysical things to happen if those observations were just inserted into the model state.  For numerical weather prediction models, errors of a few degrees in the temperature field could lead to quite different atmospheric regimes, particularly if they coincide with the right moisture amounts to spawn (or not) convection.  As a result, there are routines built into the software to avoid these sorts of things, and to keep large changes of state from happening too quickly, simply because the models are better predictors with these routines than without.  Even after this explanation, the audience seemed very uncomfortable with the idea of “tossing out observations”.

This served as a reality check for me, a reminder that there is a real divide between mathematicians that create methods for solving problems, and the scientists that use them.  That’s the reason that I write on this blog, but the reminder was useful nonetheless.