Weather or Not

Severe Outflow by R. Edwards

Human Weather Forecasting in an Automation Era, Part 3: Garbage In, Garbage Out

by Leave a Comment

This short series (go to Part 1 or Part 2) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

Objective verification of forecasts will remain hugely important, and the authors duly note that. But one factor not discussed (perhaps due to space limitations?) is the quality of the verification data. That matters…perhaps not to bureaucrats, who tend to overlook components of the verification sausage that provide context. But flawed verification datasets give you flawed verification numbers, even if the calculations are completely mathematically correct!

As someone who has analyzed and examined U.S. tornado, wind and hail data for most of my career, and published some research rooted in it, I can say two things with confidence:
1. It’s the most complete, precise and detailed data in the world, but
2. Precision is not necessarily accuracy. The data remain suffused with blobs of rottenness and grossly estimated or even completely fudged magnitudes, potentially giving misleading impressions on how “good” a forecast is.

How? Take the convective wind data, for example. More details can be found in my formally published paper on the subject, but suffice to say, it’s actually rather deeply contaminated, questionably accurate and surprisingly imprecise, and I’m amazed that it has generated as much useful research as it has. For example: trees and limbs can fall down in severe (50 kt, 58 mph by NWS definition) wind, subsevere wind, light wind, or no wind at all. Yet reports of downed trees and tree damage, when used to verify warnings, are bogused to severe numeric wind values by policy (as noted and cited in the paper). A patently unscientific and intellectually dishonest policy!

For another example, estimated winds tend to be overestimates, by a factor of about 1.25 in bulk, based on human wind-tunnel exposure (same paper). Yet four years after that research published, estimated gusts continue to be treated exactly like measured ones for verification (and now ML-informing) purposes. Why? Either estimated winds should be thrown out, or a pre-verification reduction factor applied to account for human overestimation. The secular increase in wind reports over the last few decades since the WSR-88D came online also should be normalized. That’s the far more scientifically justifiable approach than using the reports as-is, with no quality control nor temporal detrending.

For one more example, which we discussed just a little in the paper, all measured winds are treated the same, even though an increasing proportion come from non-AWOS, non-ASOS, non-mesonet instruments such as school and home weather stations. These are of questionable scientific validity in terms of proper exposure and calibration. The same can be said for storm-chaser and -spotter instrumentation, which may not be well-calibrated at a base level, and which may be either handheld at unknown height and exposure, or recording the slipstream if mounted on a vehicle.

Yet all those collectively populate the “severe” gust verification datasets also are used for training machine-learning algorithms — to the extent that actual, measured winds with scientific-grade, calibrated, verifiably properly sited instruments are a tiny minority of reports. With regard to wind reports, national outlooks, local warnings, and machine-learning training data use excess, non-severe wind data for verification, but because they all do, comparisons between them still may be useful, even if misleading.

Several of us severe-storms forecasters have noticed operationally that some ML-informed algorithms for generating calibrated wind probabilities put bull’s-eyes over CWAs and small parts of the country (mainly east) known to heavily use “trees down” to verify warnings, and that have much less actual severe thunderstorm wind (based on peer-reviewed studies of measured gusts, such as mine and this one by Bryan Smith) than the central and west. This has little to do with meteorology, and much to do with inconsistent and unscientific verification practices.

To improve the training data, the report-gathering and verification practices that inform it must improve, and/or the employers of the training data must apply objective filters. Will they?

This concludes the three-part series stimulated by Neil’s excellent paper. Great gratitude goes to Neil and his coauthors, and the handful who ever will read this far.

Reflections on a Quarter Century of Storm Forecasting

by Leave a Comment

As of last week, I have been forecasting and researching severe storms (in SELS-Kansas City and its Norman successor) for 25 years, not counting prior time at NHC and NSSL. That’s 1/4 century of living the dream of a tornado-obsessed kid. Much has transpired professionally and personally in that time span, most of it decidedly for the better. The only negative is that I’m a quarter-century older. Give how little I knew then compared to now, and how little I knew about how little I knew, maybe the geezers of my youth were right, in that youth is wasted on the young.

The science of severe-weather prediction has advanced markedly. More is understood about the development and maintenance of severe storms than ever before. Numerical models also are better than ever, yet still riddled with flaws known to forecasters that belie their hype as panaceas. Most weather media, social media weather pundits outside front-line forecasters, and far too many Twitter-active pure researchers and grad students exhibit naivete and ignorance about both the flaws of models in applied use, and the still-urgent need for humans in forecasting (yes, forecasting, not just so-called “decision support services” a.k.a. DSS).

Fortunately, most of those who actually do the job — the experienced severe-storms-prediction specialists who are my colleagues — know better, and incorporate both the science and art (yes, art!) of meteorology into forecasting, to varying extents. Yet pitfalls lie in our path in forms of several interrelated ideas:

    * Automation: Even if the human forecast is better at a certain time scale, at what point does the bureaucracy (beholden to budget, not excellence) decide the cost-benefit ratio is worth losing some forecast quality to replace humans with bots that don’t take sick leave, join unions, nor collect night differential? I wrote in much more detail about this two years ago, and that discussion touches upon some of what I am re-emphasizing below. Please go back and read that if you haven’t already.

    * Duty creep with loss of diagnostic-understanding time: Cram more nickel-and-dime, non-meteorological side duties into the same time frames with the same staffing levels, a social-media nickel this year, a video-briefing dime the next, and something must give. In my experience, that is analysis and understanding, which in an ironically self-fulfilling way, stagnates human forecast skill (and more importantly, sacrificing concentration and situational understanding) whilst allowing models to catch up. Knowing how bureaucracy works, I suspect this is by design.

    * Mission sidetracking – “DSS” including customized media and social-media services: I don’t deny the importance of DSS; in fact I support it! Outreach is good! Yet DSS should not be done by the full-time, front-line forecasters who ideally need to be laser-focused on meteorological understanding when on duty, and making forecasts the most excellent possible. DSS should be a separate and parallel staffing with social-science-trained specialists in outreach everywhere DSS is required. Otherwise, quality above what the models can provide (which still is possible, especially on Day-1 and day-2, and in complex phenomena like severe and winter storms) will be lost prematurely and unnecessarily.

    * Loss of focus — see the last two bullets: A growing body of psychological literature resoundingly debunks the notion of “multitasking”. We lose focus and delay or dilute accomplishment when concentration is broken and interruptions occur. Management should be focusing on reducing, not increasing, distractions and interruptions on the forecast desk. Forecast quality and human lives are at stake.

    * De-emphasis of science in service: Physical and conceptual understanding matter in the preparation of consistently high-quality forecasts — especially on the complicated, multi-variate area of severe local storms. These are not day-5 dewpoint grids, and this is why my workplace has published more scientific research than any other publicly funded forecasting office, by far. Tornadoes, severe hail and thunderstorm winds are highly dependent on time and space overlaps of multiple kinds of forcings that models still do not often handle well, partly because of the “garbage in, garbage out” phenomenon (input observations are not dense enough), partly due to imperfect model physics and assimilation methods. Severe-storms specialists must have both self-motivation and continued support from above to understand the science — not only by getting training and reading papers, but by writing papers and performing research!

    * Model-driven temptation to complacency: This is a form of Snellman’s meteorological cancer. I wrote about some of these topics here 13 years ago in far more detail, under the umbrella of ensemble forecasting. Please read that discussion! I see no need so far to amend any of it, except to add thoughts about focus and concentration (above). If forecasters don’t think they can improve on a model, even if they really can, or just don’t feel like making effort to do so amidst other demands for time, they’ll just regurgitate the output, at which point their jobs can (and probably should!) be automated.

    * Meddling in the mission by distant, detached bureaucratic ignoramuses. Schism between upper-management assumptions and real front-line knowledge is a common theme across all governmental and corporate bureaucracies, and is nothing new across generations. In my arena, it manifests as lack of understanding and appreciation for the difficulty and complexity of the work, and in the difference in respecting the absolutely urgent need for direct, devoted, focused human involvement. The very first people with whom policy-makers should discuss severe-storms-prediction issues are the front-line severe-storms forecasters — that is, if knowledge and understanding matter at all in making policy.

At this stage of my career, I’m neither an embittered old cynic nor a tail-wagging puppy panting with naive glee. I never was the latter and I intend not to turn into the former. Instead I observe and study developments in a level-headed way, as both an idealist and a realist, assess them with reason and logic, and report about them with brutal honesty. I doing so, I’ll say that there is cause for both optimism and pessimism at this critical juncture. I’ve covered the pitfalls (pessimism) already.

How can optimism be realized? It’s straightforward, though not easy. We must continue to grow the science, emphasize the human element of physical and conceptual understanding (including the still-important role of human understanding and the art of meteorology) in such complex and multivariate phenomena, use ever-improving (but still highly imperfect!) models as tools and not crutches, study and learn every single day, minimize distractions/disruptions, and most of all, focus on and fight for excellence!

I’m now decidedly closer to retirement than to the start of my career. Yet you can count on this: you won’t see me coast, nor go FIGMO, nor be merely “good enough for government work”! Such behavior is absolutely unacceptable, pathologically lazy, morally wrong, and completely counter to my nature. The passion for atmospheric violence still burns hot as ever.

Excellence is not synonymous with perfection, and the latter is impossible anyway. I will issue occasional bad forecasts, and I hope, far more great ones. Regardless of the fickle vagaries of individual events, I must start each new day for what it is — a different challenge ready to be tackled, compartmentalized unto itself, not the same as the great or crappy forecast of the previous shift. I must settle for nothing less than consistency of excellence in performance, lead the next generation by example in effort, and advance the science further. I’ll be pouring the best reasoning I know into each forecast, even if that is necessarily imperfect and incomplete. I’ll be doing research and writing more papers. I’ll be educating and speaking and writing and raising awareness on severe-storms topics, trying to pass understanding on to both users of the forecasts and forecasters of the future.

I’m paid well enough, and the taxpayer deserves no less than excellence in return for his/her investment in me. That is my driven purpose in the years remaining in full-time severe-weather forecasting.

Forecasting on the Edge

by 1 Comment

The Norman area sits on the edge of a possible heavy snow and/or sleet and/or freezing rain event for Tuesday. Which is it and how much, categorically? My answer as of midnight Sunday night/Monday morning: Still too soon to say! Anyone who tries to nail any spot down to a specific amount, or a narrow range (like, say, 5-6 inches) this soon is full of BS, and should be trusted no more than a used-car dealer in Vegas. Neither the human forecasters nor the models are that good yet.

While I have looked at some more recent forecasts, and have a decent grasp of the general scenario, I’ll first post and link to some SREF (short-range ensemble forecast) panels that I had a chance to grab yesterday afternoon from the morning’s run. They illustrate the difficulty faced by winter-weather forecasters really well!

[For the uninitiated, the SREF is a 21-member package of various numerical models. I don’t have room to explain it in detail here; but this site has a good summary of SREF and a big variety of forecast charts.]

The above forecast is the maximum value (basically, at any spot on the map) for the total liquid (melted) equivalent precipitation during 6Z-18Z (midnight-noon) Tuesday. Take every one of the 21 models, find its max precip, and plot that value, and basically that’s what this is–a heaviest-case scenario. Notice that most of the heaviest precip forecasts are near (but mainly south and east) of Norman. Similar forecasts ending later during the day are not quite so large around Norman. Again, this is liquid amount–not the equivalent of snow. For snow…

This forecast is for the average snowfall in inches at any given spot on the map for the 12 hours ending later that day (21Z or 3 p.m.). The times are offset some because of an expected change to snow in Norman sometime during that block of time, and as forecaster, I think this time selection probably will capture most (not necessarily all) of the snow event. Notice Norman is on the opposite edge of the heaviest snow belt from where it was with respect to the heaviest total precip accumulation ending a few hours before. Hmmm…so if we’re on one edge of melted equivalent and another of snow, where and when is the transition?

This forecast shows the average position of the freezing line southeast of Norman by 12Z (6 a.m.), but quite a bit of spread off that mean in the extreme positions (dotted and dashed lines). That freezing line matters hugely for what kind of precip we’ll have! The most likely precip type (out of all probabilities) shows rain east, a mix of sleet and freezing rain overhead and nearby, and snow just to the NW. At 12Z, we’re on the edge of a lot of things that only need to be a little bit wrong to trash the hell out of any forecast that’s too specific! Assuming the dominant forecast is accurate, this trend shifts east over us during the day to render all snow…

Now, by 21Z (3 p.m.), we’re pretty confident that it’s snow, being deep into the blue area, with all the models’ freezing lines past us, and (if you’ve also looked at forecast soundings, which I’m not showing) an understanding that it’s too cold aloft for sleet or freezing rain. But the critical issue is: when does that freezing line go past us, and how far behind it is the air above the surface warm enough to yield freezing rain or sleet before the change-over? On a national scale, it looks puny. Make a spot forecast, and just a few hours one way or another makes a huge difference.

Now let’s look at what we call “plume diagrams” — because they often look like plumes from a chimney. They’re actually spot forecasts of accumulated totals for the time period, in this case precip amounts for Norman, generated by the very same set (ensemble) of models. Each line represents one model in the set (colored by numbers at the top, so you can follow your favorite models for the situation, if you have some). The average forecast is black, with dots every 6 hours. The timeline goes from earlier at left to later at right…

This is the melted precip total, regardless of the form. We are in a drought, and this is what matters most in a hydrologic sense anyway, so we’ll look at precip totals first. Notice how the models generally agree well on when we’ll get the most precip–the steep ramp-ups between 6Z (midnight) and 21Z (3 p.m.) Tuesday. But they disagree vastly on how much (from less than a quarter inch to almost an inch and a half). Even if this were all rain, we would have one hell of a time forecasting how much…and we haven’t even looked at the precip type yet. Let’s do that!

This is the ensemble of models for accumulated freezing rain (ice). They’re all over the place too, and many of the same models that are heavier with liquid rain are lighter with ice. The green models set isn’t even there; none of those are forecasting freezing rain. That says there’s a lot of disagreement on when the transition happens with respect to how much is rain versus ice, if there’s any ice! Again, this wide variation is just in one spot (Norman)…not even considering the potential for much bigger or smaller amounts just E or W of here. Are you sure you even want to see sleet forecasts? Well, if you don’t, stop now, because that’s what’s next…

Oh, joy. Sleet forecasts are all over the place too. Not only that, a few models have high accumulations of rain, ice and sleet, a few are low on them all, and the rest vary greatly between which will be dominant, and by how much. Only the green members, which are all fairly dry across the board, seem consistent. It’s enough to make a forecaster with little patience for uncertainty throw his hands up and walk away in abject frustration. But wait! There’s more…namely, the one precip type that it seems everybody demands to know down to the inch: snow…

This model set says Norman will get anywhere from nothing to a foot, but with a low average of just above 2 inches. If those two blue models are onto something that the others are missing, millions of dollars in snow-plowing and salting expenses might be justified. If any of the models are right in forecasting a heavy snow band of, say, 16 inches someplace else, but are wrong on the location, we could get a lot more in Norman than the highest model predicts. Or, we hype it up, get nothing, all that road salt was laid down for naught, and the local governments are quite upset. Lots at stake here for the public, emergency managers, school systems, law enforcement, media, and your credibility as a forecaster…after a wildly uncertain period of rain and/or sleet and/or ice!

We’ve got quite a forecasting dilemma here. Can you see? It’s tempting to go with the averages as a hedge against a huge forecast error; but what if the extreme-upper model turns out to be right in Norman itself? Again, history tells us that sometimes, especially in the middle of a narrow and badly forecast snow band, even the extreme solution wasn’t extreme enough. Other times, the lowest solution wasn’t low enough, because we get barely a dusting, while either Slaughterville or Del City, each 15 miles away, gets over a foot. These things have happened before, and the forecaster needs to keep historical similarities in mind too.

Based on all that information, about the best we can say is that cold rain probably will change to snow, with a period of assorted freezing rain and sleet possible (but not for sure) sandwiched in between. How much? Too uncertain to call. That’s the honest answer.

At least the strong consensus is that snow, if any, starts after the rain and sleet or freezing rain. That’s not much consolation if Aunt Matilda is pestering you for an exact snow amount and when it will happen; and you just can’t explain these crazy uncertainties to her without sounding like a waffling, blathering know-nothing whose parents wasted money on your meteorology degree.

Uh oh…what’s this? After all that weeping and gnashing of teeth, let’s pretend a new SREF package has come in and the forecasts of many of the individual models have flip-flopped around like a fish out of water. Trends are up with some, down with others, earlier with some, later with others. As your local teenager might type in a text message, “OMG WTF!!!”.

If you are a forecaster, what do you do now? With all those mixed signals, and a historical precedent for everything from nothing to 15+ inches, this scenario can drive you to the brink of incoherently blubbering lunacy, and beyond.

One way to keep sane is give up, stop thinking about it, and forecast some average default, which of course a machine can do without your help. Don’t come crying to me, then, when you lose your job to that machine!

Another way to become more certain and confident in a forecast, as well as maintain sanity, is through strong physical understanding of the situation, which comes from a combination of education (school and self), training, experience, analytic skill, understanding the models, and the continuing motivation to keep up with it all. This “learned path” is not foolproof, but it’s an insurance policy against consistent failure, and one that sometimes pays off big in correctly forecasting an extreme event that lazy, “model-hugging” forecasters will miss. [About five years ago, I discussed the future of human forecasting in detail here. ]

In short, the forecaster must first diagnose what’s going on now, both to judge how the models are performing right from the start, and more importantly, to form a 4-dimensional conceptual model of the ongoing atmosphere in his own head. This means analysis–including hand analysis of surface and upper air charts–which takes time to do with due accuracy and attention to detail. A forecaster who tries to predict the future without a thorough understanding of the present is negligent, and prone to spectacular failure.

Then comes assorted model guidance. The SREF package of 21 models has far more ways to display output from them than I’ve shown here. For a truer appreciation, go to the SREF website and look in detail at everything it contains. The exercise, done carefully, will take an hour or more. Then comes operational models outside SREF, of which there are several. The variety of guidance available to the forecaster these days is dizzying. Truly I declare, there’s hardly time to look at a substantial fraction of it, much less all. It really is informational overload.

The ability to sift out the irrelevant and distill the pertinent in weather prediction is an uncommon skill, one gained mainly through experience. Even then, in the face of inflexible time deadlines, it’s easy for even the best and sharpest forecasters to overlook a potentially important but small detail anywhere along the way. Make it 4 a.m. for a rotating-shift worker, and the potential for human error rises (unless, like me, you are a bonafide night-owl). It’s also possible for some forecasters–such as the model-huggers I mentioned above–to have years and years of experience doing it in a poor way (in which case 20 years of “experience” is actually 1 year of experience repeated 20 times over). Every forecaster takes at least a slightly different approach from every other.

Given all these factors, no wonder one forecaster can differ so much from the next, and one forecaster’s own predictions can vary from one day to the next.

If you are not a meteorologist, have mercy on your friendly neighborhood forecaster. Don’t get upset with the meteorological prognosticators for being unsure, or changing their minds often, or giving you a wide range or possibilities, or differing a lot. And when your local forecasters get a winter-storm prediction right–or even close–heap laud profusely upon them, for they have accomplished an extraordinarily difficult feat. Remember: uncertainty is part of the deal. It is unavoidable, if a forecaster is being honest with himself and with you. The forecasters who are hedging or talking in probabilities are because it’s the smartest approach, the right approach.