Weather

AI in Weather Forecasting (Not the Last)

July 19, 2023 by tornado Leave a Comment

This is the first, likely not last, entry in this here ol’ BLOG that will be devoted to the use of artificial intelligence (AI), including machine-learning output, in forecasting. If you want scientifically vetted, peer-reviewed documentation of some of what I’m talking about, check out:
* Random forests for short-fused applications
* Using random forests to derive day-2 probabilities from CAM ensembles
* Random forests at various severe-day periods

Both public- and private-sector AI usage and claims are presented in this fairly presented Washington Post article.

As someone who uses, and oversees the use of, a set of AI-informed tools as a part of the daily forecast regimen, I’ll state this is not something to be afraid of, nor to lazily and mindlessly use as a crutch.

To address the fear factor—a lot of that is rooted in the black-box aspect Schumacher has discussed. He and other university and lab researchers (such as Loken) have published peer-reviewed papers (see above) discussing how their methodologies work, in reproducible ways. They’re in AMS journals, to which we in NWS have ready access. So don’t be afraid, be informed.

Admittedly it’s harder to be informed on the private-sector black boxes, because they are unscientific in the sense of not being reproducible and peer reviewed. They’re hidden behind a “proprietary” shroud of secrecy—the very antithesis of science! All you see are results—typically only the cherry-picked best, of course, with perhaps a token minor failure if the presenter is allowed to reveal such. But we don’t really know fully where those are consistently weak, due to (again!) lack of peer-reviewed documentation with reproducible methodologies, analyses and results. Caveat emptor!

To the second point (tools, not crutches), this is another in a long line of beneficial advancements in improving the forecasts overall. They add confidence, for example, to SPC day-3+ areas, but the forecaster does not just mindlessly “draw around them”. Meteorological reasoning still must be used, and discussed, by the outlook forecaster. The more uncommon a scenario is, the less the machine learning is trained on situations like them, and the more they’ll miss, even in an ensemble sense. Bulk statistical verification of AI-based forecasts may look great, but smooths over the outliers, which tend to be disproportionately destructive and important. And there’s still the “garbage in garbage out” factor, mainly due to lack of high-density upper-air obs here and even surface obs in much of the world, that renders uncertainty and output spread in ensembles right from the start.

AI/ML-informed models are imperfect, especially for the rare events that most depend on our own physical understanding as meteorologists to tease out. Lose that understanding, and we might as well be automatons regurgitating models. That might be “good enough for government work” most of the time, maybe even accidentally excellent sometimes.

Yet for anyone tempted to steer that direction, in a personal or policy sense, I offer these sobering and disturbing words:

Air France 447.

Human Weather Forecasting in an Automation Era, Part 3: Garbage In, Garbage Out

August 28, 2022 by tornado Leave a Comment

This short series (go to Part 1 or Part 2) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

Objective verification of forecasts will remain hugely important, and the authors duly note that. But one factor not discussed (perhaps due to space limitations?) is the quality of the verification data. That matters…perhaps not to bureaucrats, who tend to overlook components of the verification sausage that provide context. But flawed verification datasets give you flawed verification numbers, even if the calculations are completely mathematically correct!

As someone who has analyzed and examined U.S. tornado, wind and hail data for most of my career, and published some research rooted in it, I can say two things with confidence:
1. It’s the most complete, precise and detailed data in the world, but
2. Precision is not necessarily accuracy. The data remain suffused with blobs of rottenness and grossly estimated or even completely fudged magnitudes, potentially giving misleading impressions on how “good” a forecast is.

How? Take the convective wind data, for example. More details can be found in my formally published paper on the subject, but suffice to say, it’s actually rather deeply contaminated, questionably accurate and surprisingly imprecise, and I’m amazed that it has generated as much useful research as it has. For example: trees and limbs can fall down in severe (50 kt, 58 mph by NWS definition) wind, subsevere wind, light wind, or no wind at all. Yet reports of downed trees and tree damage, when used to verify warnings, are bogused to severe numeric wind values by policy (as noted and cited in the paper). A patently unscientific and intellectually dishonest policy!

For another example, estimated winds tend to be overestimates, by a factor of about 1.25 in bulk, based on human wind-tunnel exposure (same paper). Yet four years after that research published, estimated gusts continue to be treated exactly like measured ones for verification (and now ML-informing) purposes. Why? Either estimated winds should be thrown out, or a pre-verification reduction factor applied to account for human overestimation. The secular increase in wind reports over the last few decades since the WSR-88D came online also should be normalized. That’s the far more scientifically justifiable approach than using the reports as-is, with no quality control nor temporal detrending.

For one more example, which we discussed just a little in the paper, all measured winds are treated the same, even though an increasing proportion come from non-AWOS, non-ASOS, non-mesonet instruments such as school and home weather stations. These are of questionable scientific validity in terms of proper exposure and calibration. The same can be said for storm-chaser and -spotter instrumentation, which may not be well-calibrated at a base level, and which may be either handheld at unknown height and exposure, or recording the slipstream if mounted on a vehicle.

Yet all those collectively populate the “severe” gust verification datasets also are used for training machine-learning algorithms — to the extent that actual, measured winds with scientific-grade, calibrated, verifiably properly sited instruments are a tiny minority of reports. With regard to wind reports, national outlooks, local warnings, and machine-learning training data use excess, non-severe wind data for verification, but because they all do, comparisons between them still may be useful, even if misleading.

Several of us severe-storms forecasters have noticed operationally that some ML-informed algorithms for generating calibrated wind probabilities put bull’s-eyes over CWAs and small parts of the country (mainly east) known to heavily use “trees down” to verify warnings, and that have much less actual severe thunderstorm wind (based on peer-reviewed studies of measured gusts, such as mine and this one by Bryan Smith) than the central and west. This has little to do with meteorology, and much to do with inconsistent and unscientific verification practices.

To improve the training data, the report-gathering and verification practices that inform it must improve, and/or the employers of the training data must apply objective filters. Will they?

This concludes the three-part series stimulated by Neil’s excellent paper. Great gratitude goes to Neil and his coauthors, and the handful who ever will read this far.

Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447

August 26, 2022 by tornado Leave a Comment

This short series (go to Part 1) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

The authors briefly mention the need for forecasters to avoid the temptation to get lazy and regurgitate increasingly accurate and complex postprocessed output. I’m so glad they did, agree fully, and would have hammered the point even harder. That temptation only will grow stronger as guidance gets better (but never perfect). To use an example from my workplace, perhaps in 2022 we’re arriving at the point that an outlook forecaster can draw probabilities around ensemble-based (2005 essay), ML-informed, calibrated, probabilistic severe guidance most of the time and be “good enough for government work.”

Yet we strive higher: excellence. That necessarily means understanding both the algorithms behind such output, and the meteorology of situations enough to know when and why it can go wrong, and adapting both forecast and communication thereof accordingly. How much of the improvement curve of models and output vs. human forecasters is due to human complacency, even if unconscious? By that, I mean flattening or even ramping down of effort put into situational understanding, through inattention and detachment (see Part 1).

It’s not only an effect of model improvement, but of degradation of human forecast thinking by a combination of procedurally forced distraction, lack of focused training on meteorological attentiveness, and also, to be brutally honest, culturally deteriorating work ethic. I don’t know how we resolve the latter, except to set positive examples for how, and why, effort matters.

As with all guidance, from the early primitive-equation barotropic models to ML-based output of today and tomorrow: they are tools, not crutches. Overdependence on them by forecasters, being lulled into a false sense of security by their marginally superior performance much of the time, that complacency causing atrophy of deep situational understanding, invites both job automation and something perhaps worse: missing an extreme and deadly outlier event of the sort most poorly sampled by ML training data.

Tools, not crutches! Air France 447 offers a frightening, real-world, mass-casualty example of this lesson, in another field. Were I reviewing the Stuart et al. AMS paper, I would have insisted on that example being included, to drive a subtly made point much more forcefully.

The human-effort plateau is hidden in the objective verification because the models are improving, so net “forecast verification” appears to improve even if forecasters generally just regurgitate guidance and move on ASAP to the next social-media blast-up. Misses of rare events get averaged out or smoothed away in bulk, so we still look misleadingly good in metrics that matter to bureaucrats. That’s masking a very important problem.

Skill isn’t where it should or could be, still, if human forecasters were as fully plugged into physical reasoning as their brain capacity allows. The human/model skill gap has shrunk, and continues to, only in part because of model improvements, but also, because of human complacency. Again, this won’t manifest in publicly advertised verification metrics, which will smooth out the problem and appear fine, since the model-human combination appears to be getting better. Appearances deceive!

The problem of excess human comfort with, and overreliance on, automation will manifest as one or more specific, deadly, “outlier” event forecasts, botched by adherence to and ignorance of suddenly flawed automated guidance: the meteorological equivalent of Air France 447. This will blow up on us as professionals when forecasters draw around calibrated-guidance lines 875 times with no problem, then on the 876th, mis-forecast some notorious, deadly, economically disastrous, rare event because “the guidance didn’t show it.”

That disaster will be masked in bulk forecast verification statistics, which shall be of little consolation to the grieving survivors.

Consider yourself warned, and learn and prepare accordingly as a forecaster!