Weather

Human Weather Forecasting in an Automation Era, Part 3: Garbage In, Garbage Out

August 28, 2022 by tornado Leave a Comment

This short series (go to Part 1 or Part 2) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

Objective verification of forecasts will remain hugely important, and the authors duly note that. But one factor not discussed (perhaps due to space limitations?) is the quality of the verification data. That matters…perhaps not to bureaucrats, who tend to overlook components of the verification sausage that provide context. But flawed verification datasets give you flawed verification numbers, even if the calculations are completely mathematically correct!

As someone who has analyzed and examined U.S. tornado, wind and hail data for most of my career, and published some research rooted in it, I can say two things with confidence:
1. It’s the most complete, precise and detailed data in the world, but
2. Precision is not necessarily accuracy. The data remain suffused with blobs of rottenness and grossly estimated or even completely fudged magnitudes, potentially giving misleading impressions on how “good” a forecast is.

How? Take the convective wind data, for example. More details can be found in my formally published paper on the subject, but suffice to say, it’s actually rather deeply contaminated, questionably accurate and surprisingly imprecise, and I’m amazed that it has generated as much useful research as it has. For example: trees and limbs can fall down in severe (50 kt, 58 mph by NWS definition) wind, subsevere wind, light wind, or no wind at all. Yet reports of downed trees and tree damage, when used to verify warnings, are bogused to severe numeric wind values by policy (as noted and cited in the paper). A patently unscientific and intellectually dishonest policy!

For another example, estimated winds tend to be overestimates, by a factor of about 1.25 in bulk, based on human wind-tunnel exposure (same paper). Yet four years after that research published, estimated gusts continue to be treated exactly like measured ones for verification (and now ML-informing) purposes. Why? Either estimated winds should be thrown out, or a pre-verification reduction factor applied to account for human overestimation. The secular increase in wind reports over the last few decades since the WSR-88D came online also should be normalized. That’s the far more scientifically justifiable approach than using the reports as-is, with no quality control nor temporal detrending.

For one more example, which we discussed just a little in the paper, all measured winds are treated the same, even though an increasing proportion come from non-AWOS, non-ASOS, non-mesonet instruments such as school and home weather stations. These are of questionable scientific validity in terms of proper exposure and calibration. The same can be said for storm-chaser and -spotter instrumentation, which may not be well-calibrated at a base level, and which may be either handheld at unknown height and exposure, or recording the slipstream if mounted on a vehicle.

Yet all those collectively populate the “severe” gust verification datasets also are used for training machine-learning algorithms — to the extent that actual, measured winds with scientific-grade, calibrated, verifiably properly sited instruments are a tiny minority of reports. With regard to wind reports, national outlooks, local warnings, and machine-learning training data use excess, non-severe wind data for verification, but because they all do, comparisons between them still may be useful, even if misleading.

Several of us severe-storms forecasters have noticed operationally that some ML-informed algorithms for generating calibrated wind probabilities put bull’s-eyes over CWAs and small parts of the country (mainly east) known to heavily use “trees down” to verify warnings, and that have much less actual severe thunderstorm wind (based on peer-reviewed studies of measured gusts, such as mine and this one by Bryan Smith) than the central and west. This has little to do with meteorology, and much to do with inconsistent and unscientific verification practices.

To improve the training data, the report-gathering and verification practices that inform it must improve, and/or the employers of the training data must apply objective filters. Will they?

This concludes the three-part series stimulated by Neil’s excellent paper. Great gratitude goes to Neil and his coauthors, and the handful who ever will read this far.

Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447

August 26, 2022 by tornado Leave a Comment

This short series (go to Part 1) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

The authors briefly mention the need for forecasters to avoid the temptation to get lazy and regurgitate increasingly accurate and complex postprocessed output. I’m so glad they did, agree fully, and would have hammered the point even harder. That temptation only will grow stronger as guidance gets better (but never perfect). To use an example from my workplace, perhaps in 2022 we’re arriving at the point that an outlook forecaster can draw probabilities around ensemble-based (2005 essay), ML-informed, calibrated, probabilistic severe guidance most of the time and be “good enough for government work.”

Yet we strive higher: excellence. That necessarily means understanding both the algorithms behind such output, and the meteorology of situations enough to know when and why it can go wrong, and adapting both forecast and communication thereof accordingly. How much of the improvement curve of models and output vs. human forecasters is due to human complacency, even if unconscious? By that, I mean flattening or even ramping down of effort put into situational understanding, through inattention and detachment (see Part 1).

It’s not only an effect of model improvement, but of degradation of human forecast thinking by a combination of procedurally forced distraction, lack of focused training on meteorological attentiveness, and also, to be brutally honest, culturally deteriorating work ethic. I don’t know how we resolve the latter, except to set positive examples for how, and why, effort matters.

As with all guidance, from the early primitive-equation barotropic models to ML-based output of today and tomorrow: they are tools, not crutches. Overdependence on them by forecasters, being lulled into a false sense of security by their marginally superior performance much of the time, that complacency causing atrophy of deep situational understanding, invites both job automation and something perhaps worse: missing an extreme and deadly outlier event of the sort most poorly sampled by ML training data.

Tools, not crutches! Air France 447 offers a frightening, real-world, mass-casualty example of this lesson, in another field. Were I reviewing the Stuart et al. AMS paper, I would have insisted on that example being included, to drive a subtly made point much more forcefully.

The human-effort plateau is hidden in the objective verification because the models are improving, so net “forecast verification” appears to improve even if forecasters generally just regurgitate guidance and move on ASAP to the next social-media blast-up. Misses of rare events get averaged out or smoothed away in bulk, so we still look misleadingly good in metrics that matter to bureaucrats. That’s masking a very important problem.

Skill isn’t where it should or could be, still, if human forecasters were as fully plugged into physical reasoning as their brain capacity allows. The human/model skill gap has shrunk, and continues to, only in part because of model improvements, but also, because of human complacency. Again, this won’t manifest in publicly advertised verification metrics, which will smooth out the problem and appear fine, since the model-human combination appears to be getting better. Appearances deceive!

The problem of excess human comfort with, and overreliance on, automation will manifest as one or more specific, deadly, “outlier” event forecasts, botched by adherence to and ignorance of suddenly flawed automated guidance: the meteorological equivalent of Air France 447. This will blow up on us as professionals when forecasters draw around calibrated-guidance lines 875 times with no problem, then on the 876th, mis-forecast some notorious, deadly, economically disastrous, rare event because “the guidance didn’t show it.”

That disaster will be masked in bulk forecast verification statistics, which shall be of little consolation to the grieving survivors.

Consider yourself warned, and learn and prepare accordingly as a forecaster!

Human Weather Forecasting in an Automation Era, Part 1: Situational Understanding

August 25, 2022 by tornado Leave a Comment

This short series arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read that paper first, before proceeding.

Neil Stuart always has been a thought-provoking author and discussion partner at conferences, regarding the human role in forecasting—most certainly including communication of forecasts. He lead-authored this new article in Bulletin of the AMS that fortunately is not paywalled, and shouldn’t be, given its importance. [I posit that no published atmospheric science should be paywalled, but that’s another issue!]

Yes, automation, AI, misconceptions thereabouts, IDSS, and the continuing importance of humans in the process, are all covered—quite often in contexts that matter in a social response-based, “Was*ISsy” sense. Although I found a few minor omissions or deficiencies in my read of it, this is about as good of a review article on the topic as reasonably can be done, given the limited space allotted.

Let the discussion begin! As a highly experienced, scientific severe-storms forecaster who was not involved (none currently in my unit were, to my knowledge), I’ll have a few tidbits to add too. There are so many places to go with this. I hope we do. Such discussion matters. I’ll post few early thoughts in multiple parts here.

Whither human understanding of the meteorological situation involved in a forecast? The article heavily emphasizes evolving educational requirements, and rightly so. It also alludes to the need for continuing meteorological and situational understanding on the part of forecasters (which begins, but does not end, in university education). The atmosphere offers continuing education, in its own language, throughout our careers as forecasters. It is professionally incumbent upon us to use observational data to read and understand the language of the atmosphere!

After all, regardless of how sophisticated the forecast models or post-processing are, we cannot optimally communicate a situation we do not deeply understand. Put another way, even the most eloquent and convincing communication of an unknowingly bad forecast, or incomplete understanding of the scenario, invites potentially tragic and economically costly consequences. Or as a colleague once put it, “You can spray-paint a turd gold, and it’s still a smelly turd.”

That means human diagnosis, by all available means (including hand analysis), still is important, and will remain so. Detailed, immersive situational diagnosis (subjective chart analyses, satellite imagery, radar imagery, soundings, radar-derived wind profiles, objective analyses, observational data of all available types) is absolutely foundational to every forecast scenario, but especially short-fused, multivariate, and high-impact ones, such as severe local storms and winter storms. Optimally thorough diagnosis is necessarily time-consuming and demanding of undistracted concentration in an operational setting. I’m glad the authors recognized and noted the importance of expert roles in forecast teams, and of fully attentive acuity. Institutionally, across public and private operations, we must reduce, not increase, distractions and disruptions to concentration on the forecast floor!

Given all the above, it is crucial for university and professional continuing education to maintain emphasis on meteorological situational understanding as the fundamental core of all forecasting and IDSS, and not lose sight of fundamentals in the zeal to embrace ever-more precise and prettier output. The safety of those who we serve with the forecasts is at stake!

More in forthcoming Part 2 and Part 3…