Weather or Not

Severe Outflow by R. Edwards

  • Home
  • About
  • Archives

Powered by Genesis

Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447

August 26, 2022 by tornado Leave a Comment

This short series (go to Part 1) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

The authors briefly mention the need for forecasters to avoid the temptation to get lazy and regurgitate increasingly accurate and complex postprocessed output. I’m so glad they did, agree fully, and would have hammered the point even harder. That temptation only will grow stronger as guidance gets better (but never perfect). To use an example from my workplace, perhaps in 2022 we’re arriving at the point that an outlook forecaster can draw probabilities around ensemble-based (2005 essay), ML-informed, calibrated, probabilistic severe guidance most of the time and be “good enough for government work.”

Yet we strive higher: excellence. That necessarily means understanding both the algorithms behind such output, and the meteorology of situations enough to know when and why it can go wrong, and adapting both forecast and communication thereof accordingly. How much of the improvement curve of models and output vs. human forecasters is due to human complacency, even if unconscious? By that, I mean flattening or even ramping down of effort put into situational understanding, through inattention and detachment (see Part 1).

It’s not only an effect of model improvement, but of degradation of human forecast thinking by a combination of procedurally forced distraction, lack of focused training on meteorological attentiveness, and also, to be brutally honest, culturally deteriorating work ethic. I don’t know how we resolve the latter, except to set positive examples for how, and why, effort matters.

As with all guidance, from the early primitive-equation barotropic models to ML-based output of today and tomorrow: they are tools, not crutches. Overdependence on them by forecasters, being lulled into a false sense of security by their marginally superior performance much of the time, that complacency causing atrophy of deep situational understanding, invites both job automation and something perhaps worse: missing an extreme and deadly outlier event of the sort most poorly sampled by ML training data.

Tools, not crutches! Air France 447 offers a frightening, real-world, mass-casualty example of this lesson, in another field. Were I reviewing the Stuart et al. AMS paper, I would have insisted on that example being included, to drive a subtly made point much more forcefully.

The human-effort plateau is hidden in the objective verification because the models are improving, so net “forecast verification” appears to improve even if forecasters generally just regurgitate guidance and move on ASAP to the next social-media blast-up. Misses of rare events get averaged out or smoothed away in bulk, so we still look misleadingly good in metrics that matter to bureaucrats. That’s masking a very important problem.

Skill isn’t where it should or could be, still, if human forecasters were as fully plugged into physical reasoning as their brain capacity allows. The human/model skill gap has shrunk, and continues to, only in part because of model improvements, but also, because of human complacency. Again, this won’t manifest in publicly advertised verification metrics, which will smooth out the problem and appear fine, since the model-human combination appears to be getting better. Appearances deceive!

The problem of excess human comfort with, and overreliance on, automation will manifest as one or more specific, deadly, “outlier” event forecasts, botched by adherence to and ignorance of suddenly flawed automated guidance: the meteorological equivalent of Air France 447. This will blow up on us as professionals when forecasters draw around calibrated-guidance lines 875 times with no problem, then on the 876th, mis-forecast some notorious, deadly, economically disastrous, rare event because “the guidance didn’t show it.”

That disaster will be masked in bulk forecast verification statistics, which shall be of little consolation to the grieving survivors.

Consider yourself warned, and learn and prepare accordingly as a forecaster!

More in forthcoming Part 3…

Filed Under: Weather Tagged With: analysis, automation, communication, communication skills, education, ensemble forecasting, forecast uncertainty, forecaster, forecasting, meteorology, operational meteorology, science, severe storms, severe weather, understanding, weather

Dangerous Crutch of Automation in Weather Forecasting

October 27, 2016 by tornado Leave a Comment

In the last installment of “Scattershooting”, I offered the following short take regarding the evolving role of humans on the operational forecasting process. I’m reproducing it again here for additional elaboration based on input received since:

…

“IDSS” — a bureaucratic acronym standing for “Integrated Decision Support Services” — is the pop-fad buzzword in today’s public weather services. Behind all the frivolous sloganeering (“Weather Ready Nation”) and even more needless managerial lingo bingo, is a nugget of extreme truth: the science of operational forecasting must improve its messaging and communications capabilities. But it is possible to swing the pendulum too far, to turn professional forecasters into mere weather briefers (full-time IDSS) and let “forecasts” be straight model output. That would be a degradation of service, because models still cannot forecast multivariate, greatest-hazard events in the short term (tornado risks, winter weather, hurricane behavior, etc.) as well as human forecasters who are diligently analyzing and understanding the atmosphere. Today’s tornado and hail threat is not the same animal as single-variable temperature grids, and is far, far, far, far, far more important and impactful.

There is still a huge role in the forecast process for human understanding. Understanding has both physical and empirical bases — the science and art of forecasting. Models of course are great tools — when understood and used well. However, models themselves do not possess understanding. Moreover, models do not issue forecasts; people do. A truly excellent forecast is much more than regurgitation of numerical output in a tidy package. It is an assembly of prognostic and diagnostic understanding into a form that is communicated well to customers. A forecast poorly understood cannot be communicated well. And a bad forecast that is masterfully conveyed and understood is a disservice regardless. Eloquent communication of a crappy forecast is akin to spray-painting a turd gold. It is still a smelly turd.

Solution? Get the forecast right. All else will fall into place. Credibility is top priority! As for automation, it will proceed at the pace forecasters permit their own skills to atrophy, both individually and collectively. For those unfamiliar as to how, look up the prescient meteorological prophet Len Snellman and his term, “meteorological cancer”.

…

Now the elaboration:

Patrick Marsh supplied a remarkably insightful article on the role automation and resulting ignorance played in plunging Air France Flight 447 (Rio to Paris) into the Atlantic: Crash: How Computers Are Setting Us Up for Disaster. Please read that before proceeding; somewhat long but certainly riveting, the article should be well worth your time.

This has important ramifications to hazardous-weather forecasting too. It fits in nicely with a lot of concepts in meteorology, as a whole, about which I have been thinking and writing for many years (e.g., here, and here, and here, and here), and as have even more-accomplished scientists (e.g., here). Let’s hope the “powers that be” read these lessons and pay attention.

In the meantime, even though it has nothing directly to do with severe midlatitude storms or other hazardous-weather forecasting done out of Norman, the Air France crash article offers such a powerful and plainspoken lesson that it could and arguably should be included in all new-forecaster training materials. My unnamed office has remained ahead of the automation monster and in control of it, because the forecasters see and understand its limitations on a daily basis, as with (for example) SREF calibrated probabilities and so-called “convection-allowing models” (CAMs) at fine grid spacing, such as the SSEO members and the NCAR ensemble. My office is at the forefront of how to help to develop the tool and use it in its proper role, without dangerous over-reliance; the model is the slave, not the human. Let’s keep it that way.

One way we can do so as forecasters is through continuing to provide meaningful developmental and testing input, as we have. This way we still understand the monster we help to create, and know when and how to employ its growing powers. Another, even more important way is by maintaining the high level of both baseline meteorological understanding and diagnostic situational awareness of events for which we are properly well-known and -respected. As I have stated resolutely elsewhere: A forecaster who tries to predict the future without a thorough understanding of the present is negligent, and prone to spectacular failure.

Keeping up with the science, and maintaining and improving fundamental skills, are so important. How? Reading, training and practice…reading, training and practice! Writing (research papers) helps immensely too, in literature review and assembly. Forecasting experience matters too, as long as it involves experiential learning and progress, not one year experience repeated over and over! Again, they key word and concept here is understanding. When forecasters’ core abilities atrophy from disuse and weaknesses creep in, perhaps even unrealized (as was the case on that A380 disaster), is when the atmosphere analogously will make us pay.

Nearly four decades ago (Snellman 1977, Bulletin of the AMS), visionary scientist Len Snellman foresaw this when he coined the term, “meteorological cancer”, to describe the threat of over-reliance on automated output to our ability to predict the weather. This can include the extreme and/or exceptional events, the “money events” that disproportionately cause casualties and destroy property. What is my money event? The tornado outbreak, and to some extent the derecho.

Since such events are, by nature and definition, extreme and unusual, an “average” model forecast might not get there. Predicting the mean on a day with little precedent can be horribly wrong. Sometimes the ensemble outlier is the closest solution, not the ensemble mean or median. We will need to be able to recognize in advance when that could be reasonably possible. [Emphasis on “reasonable”–not “you never know, you can’t rule it out” grade of total CYA over-emphasis on the outliers.] Our probability of recognizing the correctness of the outlier increases with our understanding, both of the ensemble and of the meteorology behind the exceptions, then in turn, our ability to diagnose when the model is veering or could veer rapidly awry, thereby nailing a once-in-career success and saving some lives to boot. Either that, or go down like that plane and suffer spectacular, career-defining failure…

We can and must prevent the latter, on both individual and institutional levels. Meteorological understanding and diagnostic ability don’t guarantee success, but they make good forecasts much more likely and consistent. When we as a science will get in trouble is when we start treating automated output as an unquestioned crutch instead of as a tool in a holistic toolbox that still involves human diagnostics and scrutiny. Is that skill being lost in the field with less-experienced forecasters who have never had to fly anything but autopilot? Will “SuperBlend” become “SuperCrutch”? If not, when will we reach that tipping point, what sort of disaster will it take to reveal such a failure, and how can we see it coming in time to change course and lift that altitude? Vague as the proposal is for now regarding specifics on forecaster duties and staffing and office-hour reductions, does “Operations and Workforce Analysis” (OWA, NWS document and NWSEO union response), for all its superficial and advertised benefits for “IDSS”, carry a dark undercurrent of swinging the automation pendulum on the forecast side too far toward that fate?

These questions and more need to be discussed. Now that truly would be (to use a piece of bureaucratic jargon) a “vital conversation”.

Filed Under: Weather Tagged With: automation, ensemble forecasting, forecaster, meteorological cancer, meteorology, numerical guidance, operational meteorology, research, science, severe storms, severe weather, situational awareness, weather forecaster

Toward More Accurate Forecast Discussions

May 22, 2014 by tornado Leave a Comment

Forecast discussions are an important way for operational meteorologists to explain the reasoning behind the forecast, and to offer additional insights to a more scientifically fluent audience than what’s necessary for public watches, warnings and forecasts. Whether it’s an “area forecast discussion” from a local office, a national center’s technical text, or a private meteorologist’s offering on a storm discussion board, these can be fantastic ways to see the reasoning behind a prediction, to learn why a forecast decision was made, and to educate oneself on the “weather whys”.

As someone who has written thousands of detailed, national forecast discussions over the years–and made more than a few mistakes doing so–I’ve come to understand that any level of timeliness, consistency, writing prowess or even forecast accuracy can be undermined by incorrect or misleading information in the text. If the audience can’t trust the truth of what they’re reading, how can they trust the forecast itself?

If we’re all literally accurate with the terminology and its usage, we’ll be not only correct, but consistent, in our communication. Forecast discussions are meant for reasonably well-educated audiences with some meteorological background–not our moms, kids or siblings. “Dumbing down” isn’t necessary. We should get the terminology and wording right, to the greatest extent our understanding allows. Then we not only communicate with truth, we exude earned authority and credibility, and sometimes, even educate the already well-educated audience.

Good science is good service; there’s no need to choose between the two. Let’s say it right! Avoid map-room jargon and literal inaccuracy while being specific about what’s happening.

Here are some common misuses of terminology from actual forecast discussions that I’ve either seen or received, and more importantly, the reasoning with solutions to improve it. I’ll be glad to add more as suggested and documented.

=============

BAD WORDING: Trough axis…ridge axis

REASONING: Redundant. A trough is an axis, by definition–in this case, an axis of low pressure or low height. A ridge is an axis of high pressure or high height. This is rather akin to saying, “cobra snake” or “copper metal”.

BETTER ALTERNATIVE(S): Trough….ridge

=============

BAD WORDING: Diffluent…diffluence

REASONING: Misspelled. The root “fluent” is preceded by the common English prefixes “di” or “con” to denote the sign of the process. This is much like “divergence” and “convergence”. Would you spell the former as “divvergence”? Of course not. So why misspell the other word as “diffluence”? [Unfortunately, this has become very common, even to the extent it has been erroneously programmed into automated spell-checkers.]

BETTER ALTERNATIVE(S): Difluent…difluence

=============

BAD WORDING: Data shows…data indicates…data offers…

REASONING: Subject-verb error. The word “data” is plural–for the singular “datum”.

BETTER ALTERNATIVE(S): Data show…data indicate…data offer…

=============

BAD WORDING: A criteria…

REASONING: Subject-verb error. The word “criteria” is plural–for the singular “criterion”.

BETTER ALTERNATIVE(S): A criterion…

=============

BAD WORDING: Narrow axis…broad axis (of moisture, temperature, CAPE, etc.)

REASONING: Physically and mathematically impossible! An axis is, at most, two-dimensional (if it curves) or one-dimensional (if straight). By definition, an axis has no width; it cannot be “narrow” or “broad” or “wide”. What we usually mean instead is…

BETTER ALTERNATIVE(S): Narrow corridor…wide corridor (or swath…area…field, etc.)

=============

BAD WORDING: Isolated tornado…isolated tornadoes

REASONING: Redundant. For forecast purposes, all tornadoes are isolated. The terms, “isolated”, “widely scattered” and “scattered”, in meteorology, deal with coverage. Isolated is <15%, widely scattered is 15-24%, scattered is 25-54%, and numerous is 55-100%. Tornadoes--even the very largest ones--cover a very tiny percent of the land area on outlook and watch scales, and most often, even on warning scales. For example, consider a major outbreak day, with multiple violent, long-tracked tornadoes--say ten of them for easy arithmetic. Let's assume the average path length and width of each tornado is 50 miles and 1/2 mile, respectively. [This is a very terrible, once-in-career type of outbreak!] That's 50*0.5*10 square miles, or 250 square miles hit by tornadoes. The average watch covers about 25,000 square miles. Let's now assume the outbreak was extremely dense, and cram all 10 of those extremely high-end tornadoes entirely inside just one average watch (unprecedented!). Even in such crazy circumstances, only 1/100 of the watch area was covered by those tornadoes: a mere one percent (1%)! That's still isolated, by definition, and if there were a category of "very isolated", it would be on the low end of that.

BETTER ALTERNATIVE(S): A tornado…a tornado or two…a few tornadoes…

=============

BAD WORDING: Deep moisture

REASONING: Misleading. Moisture is everywhere in the troposphere! Instead this usually means a deep layer of (some measure of) moisture. The layer containing that moisture can be deep or shallow. Specify that instead.

BETTER ALTERNATIVE(S): Deep, moist boundary layer…deep saturated layer…deep layer of mixing ratios at least 14 g/kg…deep layer of 50% or higher RH, etc.

=============

BAD WORDING: Moisture pooling

REASONING: Misleading and colloquial, similar to “deep moisture”. This is one case where using fewer words destroys the real meaning. Again, moisture is everywhere in the troposphere! Instead this misused phrase usually stands for a deep and horizontally restricted layer of enhanced, low-level moisture. Such a condition often occurs near boundaries. It would be more insightful and literally correct to specify the physical mechanism behind that.

BETTER ALTERNATIVE(S): Zone (or corridor or area or deep layer) of relatively rich moisture with an outflow boundary, etc.

=============

BAD WORDING: Vortice

REASONING: No such word.

BETTER ALTERNATIVE(S): Vortex

=============

BAD WORDING: Thundershower…

REASONING: No such thing.

BETTER ALTERNATIVE(S): Thunderstorm…

=============

BAD WORDING: Instability of 2,000 J/kg

REASONING: Erroneous. The writer means CAPE (measured in J/kg, unlike actual instabilities). CAPE and convective instability are not physically the same! CAPE is an indicator of convective instability; but there are some other kinds of instability–for example, Helmholtz or inertial that can happen with no CAPE. The links go to AMS glossary definitions.

BETTER ALTERNATIVE(S): CAPE of 2,000 J/kg

=============

BAD WORDING: Overrunning

REASONING: Misleading, colloquial overreach. Opinions vary on the validity of the term in very specific situations (See one essay by Monteverdi and another by Doswell), but it’s usually misused anyway. It’s more insightful to specify the actual process involved.

BETTER ALTERNATIVE(S): Frontal lift or warm advection

=============

BAD WORDING: Strong (or weak) upper-level dynamics

REASONING: Too vague, virtually meaningless. The term is so broad and overused that it encompasses almost limitless possibilities. Most (certainly not all) of the time, its context implies some form of, or contributor to, vertical motion. Be specific, not vague, about what’s happening and why–or if you don’t know, just say so. Don’t use “dynamics” as a black-box, catch-all word. Ambiguity is the enemy of understanding!

BETTER ALTERNATIVE(S): This depends on the situation. Refer directly to the actual physical process at work, whether it’s large-scale vertical motion contributions from differential vorticity advection, a thermally indirect circulation near a jet streak, mesoscale lift sources such as outflow boundaries, or storm-scale internal “dynamics” (the vertical pressure-gradient force), for example. In short, be specific.

=============

BAD WORDING: Model resolution

REASONING: (Usually) wrong usage. Resolution of a model is not the same as its grid spacing, which is usually what the writer means. However, even then, the writer often does not realize that true horizontal resolution of features occurs only at 4X or more coarser than grid spacing! This very short scientific essay does a great job of explaining the difference.

BETTER ALTERNATIVE(S): Model grid spacing (or if using the higher number and the literal meaning) model resolution

=============

BAD WORDING: DPVA causing (forcing, initiating, triggering) thunderstorms

REASONING: Misleading and false. Vertical motion related to differential vorticity advection is on the scale of cm/s or weaker. DCVA (same as DPVA in the northern hemisphere) makes the environment more thermodynamically unstable by steepening lapse rates, but a stronger source of lift is needed to initiate thunderstorms, within which air rises on the scale of tens of m/s (three orders of magnitude or more stronger than DCVA!). If an updraft is blasting through the sky at 50 m/s, half a cm/s of downward motion from DNVA won’t stop that; however there can be an indirect effect. Stable layers or subsident drying that result from DNVA can make the environment less favorable for storms.

BETTER ALTERNATIVE(S): DCVA destabilizing the environment for thunderstorms

=============

BAD WORDING: “Energy” in the context of perturbations or the instability they may cause, such as upper-level energy approaching

REASONING: Same as with “dynamics”, but worse. Energy, in many forms, is everywhere in the atmosphere! As such, this word is meaningless.

BETTER ALTERNATIVE(S): As with “dynamics”, this depends on the specific situation. Just state the actual feature or process involved. A more-informed reader is…a more-informed reader!

=============

BAD WORDING: Strong (or weak) lift or rising motion or subsidence (without specifying the source)

REASONING: Much too vague! Lift or sinking occurs on every scale of the atmosphere, from countless causes, and on a huge range of speeds and scales.

BETTER ALTERNATIVE(S): State the actual cause or process. For example, frontal lift…isentropic lift…sinking motion in cold advection, or whatever other actual process is behind the lift or subsidence.

=============

BAD WORDING: Shortwave approaching…

REASONING: Incomplete. Waves have both ridge and trough components. Almost always, the writer means shortwave trough, so be specific and say so.

BETTER ALTERNATIVE(S): Shortwave trough approaching…, or if you truly mean the entire wave (ridge and trough), stick with “shortwave”.

=============

BAD WORDING: Deviate motion

REASONING: Misspelling and/or usage error. Deviate is a verb. Deviant is the adjective form.

BETTER ALTERNATIVE(S): Deviant motion

=============

BAD WORDING: EF3 tornado intensity (or any other EF rating with the word “intensity”)

REASONING: Tornado rating is not the same as tornado intensity. The EF scale (or before it, the F Scale) is a way of rating tornadoes based on damage–or estimating a probable intensity at the damage point based on damage. However, most damage indicators (DIs) do not go up to EF5, and often there is either no DI or a weak DI. As such, the actual intensity of a tornado at most points cannot be specified from damage alone! The best we can do is estimate–often very crudely at that. Furthermore, many factors go into damage rating, wind intensity being only one of them (others including debris impact, construction methods, exposure, and subjective judgment of the surveyor). In short, the true intensity of most tornadoes is unknown, regardless of damage.

BETTER ALTERNATIVE(S): EF3 tornado damage–or, if referring specifically to wind speeds in the context of the EF scale standards or measured winds, simply EF3 wind speeds.

=============

BAD WORDING: It is important to note that…it should be noted that… and other “It…that…” statements

REASONING: I see this more often with scientific papers (!), but sometimes in forecast discussions also. These phrases are unnecessary and almost always can be removed completely, saving space. Look, would you note something unimportant? Would you note something that should not be noted? Of course not! So what’s the point? Don’t bother with the wasted words. For more on this needless wording, see the book Eloquent Science by David Schultz, specifically Table 10.2 and adjoining discussion, for a list of these phrases, and alternatives if desired. Again, most of the time you simply can remove the phrase.

BETTER ALTERNATIVE(S): Delete. Start the sentence with whatever came after that phrase–capitalizing the first word, of course.

=============

Filed Under: Weather Tagged With: accuracy, credibility, forecast, forecast discussion, forecaster, forecasting, inaccuracy, jargon, literal, maproom jargon, meteorology, technical discussion, terminology, weather discussion, weather forecast, weather forecaster

Search

Recent Posts

  • Scattershooting 230128
  • A Thanksgiving Message
  • Human Weather Forecasting in an Automation Era, Part 3: Garbage In, Garbage Out
  • Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447
  • Human Weather Forecasting in an Automation Era, Part 1: Situational Understanding

Categories

  • Not weather
  • Photographic Adventures
  • Scattershooting
  • Weather
  • Weather AND Not
@SkyPixWeather

- January 30, 2023, 3:38 am

@cschultzwx @TwisterKidMedia So many holds don’t get called. That looked quite familiar. I know this as a Cowboys/Micah Parsons fan.
h J R
@SkyPixWeather

- January 30, 2023, 3:27 am

@TwisterKidMedia @sdantwx Worst officiating I’ve ever seen was in a college game too, and Andrew should know this one. https://t.co/pC5fTkFFrF
h J R
@SkyPixWeather

- January 30, 2023, 3:20 am

@TwisterKidMedia @tempestchasing I still don’t understand fully WTH happened w/the “unheard whistle” clock debacle and play that wasn’t. That was bizarre.
h J R

Blogroll

  • CanadianTexan
  • Chuck's Chatter
  • Cliff Mass Weather & Climate
  • Digital Photography Review
  • DMN Dallas Cowboys BLOG
  • Dr. Cook's Blog
  • Dr. JimmyC
  • E-journal of Severe Storms Meteorology
  • Eloquent Science
  • Image of the Week
  • Jack's Cam Wall
  • Jim LaDue View
  • Laura Ingraham
  • MADWEATHER
  • Michelle Malkin
  • Photography Attorney
  • Severe Weather Notes
  • SkyPix by Roger Edwards
  • Tornatrix
  • With All My Mind

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org