analysis

Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447

August 26, 2022 by tornado Leave a Comment

This short series (go to Part 1) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

The authors briefly mention the need for forecasters to avoid the temptation to get lazy and regurgitate increasingly accurate and complex postprocessed output. I’m so glad they did, agree fully, and would have hammered the point even harder. That temptation only will grow stronger as guidance gets better (but never perfect). To use an example from my workplace, perhaps in 2022 we’re arriving at the point that an outlook forecaster can draw probabilities around ensemble-based (2005 essay), ML-informed, calibrated, probabilistic severe guidance most of the time and be “good enough for government work.”

Yet we strive higher: excellence. That necessarily means understanding both the algorithms behind such output, and the meteorology of situations enough to know when and why it can go wrong, and adapting both forecast and communication thereof accordingly. How much of the improvement curve of models and output vs. human forecasters is due to human complacency, even if unconscious? By that, I mean flattening or even ramping down of effort put into situational understanding, through inattention and detachment (see Part 1).

It’s not only an effect of model improvement, but of degradation of human forecast thinking by a combination of procedurally forced distraction, lack of focused training on meteorological attentiveness, and also, to be brutally honest, culturally deteriorating work ethic. I don’t know how we resolve the latter, except to set positive examples for how, and why, effort matters.

As with all guidance, from the early primitive-equation barotropic models to ML-based output of today and tomorrow: they are tools, not crutches. Overdependence on them by forecasters, being lulled into a false sense of security by their marginally superior performance much of the time, that complacency causing atrophy of deep situational understanding, invites both job automation and something perhaps worse: missing an extreme and deadly outlier event of the sort most poorly sampled by ML training data.

Tools, not crutches! Air France 447 offers a frightening, real-world, mass-casualty example of this lesson, in another field. Were I reviewing the Stuart et al. AMS paper, I would have insisted on that example being included, to drive a subtly made point much more forcefully.

The human-effort plateau is hidden in the objective verification because the models are improving, so net “forecast verification” appears to improve even if forecasters generally just regurgitate guidance and move on ASAP to the next social-media blast-up. Misses of rare events get averaged out or smoothed away in bulk, so we still look misleadingly good in metrics that matter to bureaucrats. That’s masking a very important problem.

Skill isn’t where it should or could be, still, if human forecasters were as fully plugged into physical reasoning as their brain capacity allows. The human/model skill gap has shrunk, and continues to, only in part because of model improvements, but also, because of human complacency. Again, this won’t manifest in publicly advertised verification metrics, which will smooth out the problem and appear fine, since the model-human combination appears to be getting better. Appearances deceive!

The problem of excess human comfort with, and overreliance on, automation will manifest as one or more specific, deadly, “outlier” event forecasts, botched by adherence to and ignorance of suddenly flawed automated guidance: the meteorological equivalent of Air France 447. This will blow up on us as professionals when forecasters draw around calibrated-guidance lines 875 times with no problem, then on the 876th, mis-forecast some notorious, deadly, economically disastrous, rare event because “the guidance didn’t show it.”

That disaster will be masked in bulk forecast verification statistics, which shall be of little consolation to the grieving survivors.

Consider yourself warned, and learn and prepare accordingly as a forecaster!

Human Weather Forecasting in an Automation Era, Part 1: Situational Understanding

August 25, 2022 by tornado Leave a Comment

This short series arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read that paper first, before proceeding.

Neil Stuart always has been a thought-provoking author and discussion partner at conferences, regarding the human role in forecasting—most certainly including communication of forecasts. He lead-authored this new article in Bulletin of the AMS that fortunately is not paywalled, and shouldn’t be, given its importance. [I posit that no published atmospheric science should be paywalled, but that’s another issue!]

Yes, automation, AI, misconceptions thereabouts, IDSS, and the continuing importance of humans in the process, are all covered—quite often in contexts that matter in a social response-based, “Was*ISsy” sense. Although I found a few minor omissions or deficiencies in my read of it, this is about as good of a review article on the topic as reasonably can be done, given the limited space allotted.

Let the discussion begin! As a highly experienced, scientific severe-storms forecaster who was not involved (none currently in my unit were, to my knowledge), I’ll have a few tidbits to add too. There are so many places to go with this. I hope we do. Such discussion matters. I’ll post few early thoughts in multiple parts here.

Whither human understanding of the meteorological situation involved in a forecast? The article heavily emphasizes evolving educational requirements, and rightly so. It also alludes to the need for continuing meteorological and situational understanding on the part of forecasters (which begins, but does not end, in university education). The atmosphere offers continuing education, in its own language, throughout our careers as forecasters. It is professionally incumbent upon us to use observational data to read and understand the language of the atmosphere!

After all, regardless of how sophisticated the forecast models or post-processing are, we cannot optimally communicate a situation we do not deeply understand. Put another way, even the most eloquent and convincing communication of an unknowingly bad forecast, or incomplete understanding of the scenario, invites potentially tragic and economically costly consequences. Or as a colleague once put it, “You can spray-paint a turd gold, and it’s still a smelly turd.”

That means human diagnosis, by all available means (including hand analysis), still is important, and will remain so. Detailed, immersive situational diagnosis (subjective chart analyses, satellite imagery, radar imagery, soundings, radar-derived wind profiles, objective analyses, observational data of all available types) is absolutely foundational to every forecast scenario, but especially short-fused, multivariate, and high-impact ones, such as severe local storms and winter storms. Optimally thorough diagnosis is necessarily time-consuming and demanding of undistracted concentration in an operational setting. I’m glad the authors recognized and noted the importance of expert roles in forecast teams, and of fully attentive acuity. Institutionally, across public and private operations, we must reduce, not increase, distractions and disruptions to concentration on the forecast floor!

Given all the above, it is crucial for university and professional continuing education to maintain emphasis on meteorological situational understanding as the fundamental core of all forecasting and IDSS, and not lose sight of fundamentals in the zeal to embrace ever-more precise and prettier output. The safety of those who we serve with the forecasts is at stake!

More in forthcoming Part 2 and Part 3…

Thoughts on the Atlanta Snow Debacle

January 30, 2014 by tornado Leave a Comment

Photo courtesy Georgia D.o.T.

A horrendous winter-weather debacle unfolded in Atlanta and a few other Southeastern cities two nights ago, far, far out of proportion to the puny total of precipitation that fell. Less then three inches of snow basically paralyzed a metro area. As I got wind of this, and as a professional meteorologist with an unnamed entity that I’m not representing here, I couldn’t help but think, “Why”?

Notice the question isn’t, “Who is to blame?”, but is the more fair and balanced, “Why?” That’s a deliberate distinction. Asking questions, trying to get to the bottom of things, and finding holes in the Integrated Warning System that lead to disasters, is not the same as blaming. If you can’t understand the difference, stop reading right now. The rest will make no sense.

Being an atmospheric scientist and forecaster by trade, my first inclination was to look at the available public- and private-sector predictions for the event. My impression was they, while not perfect, they generally were very good–including and especially the forecasts from NWS Peachtree City. Contrary to the impulsive, borderline slanderous scapegoating by the Georgia governor and city officials, with no understanding or justification of how forecasting uncertainty works, I came away with the impression that an extremely difficult forecast was performed fairly well. This story on the backlash from some private-sector meteorologists illustrates that well.

Were all the forecasts from all sources good? Probably not. James Spann, Birmingham TV meteorologist for whom I have a great deal of respect, perhaps was too hard on himself when he said, “Days like yesterday, unfortunately, are part of my job. There have been bad forecasts in the past, and there will be bad ones in the future. Football coaches don’t win every game, and we don’t get every forecast right. But, when you lose, you do deep study into what went wrong, and work to be sure it doesn’t happen again.” The undercurrent there, however, is that forecasting cannot be totally right; and to expect perfection is just plain stupid. Please understand a key part of what James stated: There will be bad ones in the future. His ethic of learning and improvement is exactly the right attitude to take–not only in weather forecasting, but in preparedness for bad weather (more on that below).

Meanwhile this quote from Nathan Deal illustrates the problem that politics causes: “If we closed the city of Atlanta and our interstate system based on maybes, then we would not be a very productive government or a city. We can’t do it based on the maybes.” News flash, Nathan: forecasting necessarily involves maybes. It’s called uncertainty, and is unavoidable in forecasting. Get used to it, and plan accordingly, instead of complaining about forecasters failing to meet impossible standards of perfection.

While those ignorant blowhard politicians thunder their hollow indignation across the TV screen and throw the local NWS under the bus, I’d like to put Peachtree City up for a medal. Although not a forecaster by trade, Marshall Shepherd offered a hugely appreciated and very well-reasoned summary of the problem, including a strong message of thanks to the forecasters. So…thank you, Marshall. You beat me to a lot of the same points, and saved this BLOG entry from being even longer than it already is.

To that, there’s little I can add regarding the forecasting. A better predictive performance with such a hard and uncommon (for them) type of event as a sub-mesoscale northern edge of a snow band–especially more than a few hours out (when a winter storm warning was in effect)–would be demanding the unreasonable. Alas, because the response to the event and the resulting impact thereof each were so ghastly (which is out of control of the meteorologists), such an award isn’t likely to happen. That’s a shame.

Response and preparedness–which are two different but interlocking facets of the Integrated Warning System–absolutely do matter! The readiness in Atlanta, both governmentally and on individual levels multiplied by hundreds of thousands, was nothing short of wretched. Even in Dallas, San Antonio and Houston–cities of roughly similar metro populations as ATL that are farther south in latitude–hasn’t experienced an ordeal like that in their winter-weather events. Could it be that even in those places, just enough folks know to cancel plans in advance when winter weather is forecast?

Vehicular traffic is horrid in Atlanta on many a fair-weather day, and just one or two well-misplaced wrecks can render the situation FUBAR. Have you looked at a road map of that place? There are hardly any gridded, straightforward alternative routes to the freeway system, which itself looks like twisted noodles. Throw hundreds of thousands of vehicles onto that nonsensically chaotic spaghetti diagram at once, and into conditions for which few of their drivers are individually experienced or prepared, with essentially no pre-treatment or treatment of roads thanks to lack of suitable equipment, material and foresight at the civic level, and voila! You get what they got. In such gridlock, with the cars that can spin their wheels going nowhere in the process, the situation goes from FUBAR for a couple of hours to unprecedented and dangerous stasis. This was preventable.

Individuals: There is individual responsibility in this! Be aware of the forecast–the very latest forecast, since they can and do and should change as the event gets closer. Pay heed. If you’re inexperienced at driving on ice and snow, then don’t drive in ice and snow. Stay at home. Leave the roads clear for those who really need to be out there. If already at work, stay there awhile and let things clear up–it’s a warm place and beats sitting in traffic the same amount of time or longer with a hundred thousand other lemmings, burning gobs of fuel, stressing over being stuck, risking hypothermia should the need to evacuate the vehicle arise, and potentially being hit by idiots sliding into you.

State and local governments: All disasters are local. This means it’s up to you to be ready–not 1-3 days before, but months before. It’s not up to forecasters to do the impossible and tell you exact snow depths tomorrow down to the block and lot. You have to make decisions based on uncertainty! Deal with it…that’s your job. Maybe it’s not “cost-effective” for a big southern city that does get pretty cold sometimes to have access to lots of sand and salt trucks, and the sand and salt to go therein, and a strategic contingency plan with short-fused, priority-driven disbursement of the vehicles and material. Fine–don’t complain, then, when the cost/benefit ratio you so carefully weighed turns out to be wrong and comes back to bite you hard. Learn from this and quit trying to play childish blame games. Cooperate across city and county and school-district borders instead of myopically operating as insular little fiefdoms; the North-Central Texas Council of Governments (“230 member governments including 16 counties, numerous cities, school districts, and special districts”) is a great template to follow! Finally, emergency management exists for a reason, and this qualifies as an emergency. Make use of that expertise already located right under your noses.

Politicians: Admit your mistakes, for once, in a very specific manner, with clearly stated plans for how you’re not going to repeat them. Quit trying to blame those least deserving of it (and as public servants, least in a position to defend themselves). The Ray Nagin school of blame-shifting should have been closed long ago.

Media: You are the mainline communicators between meteorology and the public. I have one “don’t” and a lot more “do’s” here. Don’t give mixed signals, multiple model forecasts, and other confusing messages. Do keep it clear and straightforward. Do express uncertainty, and use that to convey the “better safe than sorry” message. Do keep up with changes in the forecast, because uncertainty mostly tends to shrink as the event gets closer. Do everything possible to encourage caution, safety and preventative avoidance of the roadways in a winter-storm scenario.

Forecasters: You (we) did well, overall. Not perfectly, but under the circumstances, not bad either. We all can learn and improve from this event, as James alluded. Remember: there’s more to consistently reliable forecasting than just models. Some folks actually hand-analyze surface and upper-air charts, investigate satellite imagery, examine and modify real soundings, and perform other physically insightful diagnostics of the actual atmosphere before ever invoking the prognostic models. Thorough analysis is the difference between merely seeing and truly understanding. This is also part of Snellman’s “Forecast Funnel” approach and is time-tested. The day we let the models do our jobs for us is the day those jobs can be automated.

This event was a woeful concatenation of misfortunes: natural forecast uncertainty, unprepared and ignorant individuals times hundreds of thousands, communications failures, badly designed transportation options, terribly ill-prepared governmental entities, and the worst timing of a snow event with respect to a weekday commuting scenario. The bottom line: Winter weather is only as bad as your preparedness for it. Take heed, Atlanta, and do it better next time. Let’s all learn from this, lest it be repeated.