Weather or Not

Severe Outflow by R. Edwards

Human Weather Forecasting in an Automation Era, Part 2: Lessons of Air France 447

by Leave a Comment

This short series (go to Part 1) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

The authors briefly mention the need for forecasters to avoid the temptation to get lazy and regurgitate increasingly accurate and complex postprocessed output. I’m so glad they did, agree fully, and would have hammered the point even harder. That temptation only will grow stronger as guidance gets better (but never perfect). To use an example from my workplace, perhaps in 2022 we’re arriving at the point that an outlook forecaster can draw probabilities around ensemble-based (2005 essay), ML-informed, calibrated, probabilistic severe guidance most of the time and be “good enough for government work.”

Yet we strive higher: excellence. That necessarily means understanding both the algorithms behind such output, and the meteorology of situations enough to know when and why it can go wrong, and adapting both forecast and communication thereof accordingly. How much of the improvement curve of models and output vs. human forecasters is due to human complacency, even if unconscious? By that, I mean flattening or even ramping down of effort put into situational understanding, through inattention and detachment (see Part 1).

It’s not only an effect of model improvement, but of degradation of human forecast thinking by a combination of procedurally forced distraction, lack of focused training on meteorological attentiveness, and also, to be brutally honest, culturally deteriorating work ethic. I don’t know how we resolve the latter, except to set positive examples for how, and why, effort matters.

As with all guidance, from the early primitive-equation barotropic models to ML-based output of today and tomorrow: they are tools, not crutches. Overdependence on them by forecasters, being lulled into a false sense of security by their marginally superior performance much of the time, that complacency causing atrophy of deep situational understanding, invites both job automation and something perhaps worse: missing an extreme and deadly outlier event of the sort most poorly sampled by ML training data.

Tools, not crutches! Air France 447 offers a frightening, real-world, mass-casualty example of this lesson, in another field. Were I reviewing the Stuart et al. AMS paper, I would have insisted on that example being included, to drive a subtly made point much more forcefully.

The human-effort plateau is hidden in the objective verification because the models are improving, so net “forecast verification” appears to improve even if forecasters generally just regurgitate guidance and move on ASAP to the next social-media blast-up. Misses of rare events get averaged out or smoothed away in bulk, so we still look misleadingly good in metrics that matter to bureaucrats. That’s masking a very important problem.

Skill isn’t where it should or could be, still, if human forecasters were as fully plugged into physical reasoning as their brain capacity allows. The human/model skill gap has shrunk, and continues to, only in part because of model improvements, but also, because of human complacency. Again, this won’t manifest in publicly advertised verification metrics, which will smooth out the problem and appear fine, since the model-human combination appears to be getting better. Appearances deceive!

The problem of excess human comfort with, and overreliance on, automation will manifest as one or more specific, deadly, “outlier” event forecasts, botched by adherence to and ignorance of suddenly flawed automated guidance: the meteorological equivalent of Air France 447. This will blow up on us as professionals when forecasters draw around calibrated-guidance lines 875 times with no problem, then on the 876th, mis-forecast some notorious, deadly, economically disastrous, rare event because “the guidance didn’t show it.”

That disaster will be masked in bulk forecast verification statistics, which shall be of little consolation to the grieving survivors.

Consider yourself warned, and learn and prepare accordingly as a forecaster!

More in forthcoming Part 3…

Human Weather Forecasting in an Automation Era, Part 1: Situational Understanding

by Leave a Comment

This short series arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read that paper first, before proceeding.

Neil Stuart always has been a thought-provoking author and discussion partner at conferences, regarding the human role in forecasting—most certainly including communication of forecasts. He lead-authored this new article in Bulletin of the AMS that fortunately is not paywalled, and shouldn’t be, given its importance. [I posit that no published atmospheric science should be paywalled, but that’s another issue!]

Yes, automation, AI, misconceptions thereabouts, IDSS, and the continuing importance of humans in the process, are all covered—quite often in contexts that matter in a social response-based, “Was*ISsy” sense. Although I found a few minor omissions or deficiencies in my read of it, this is about as good of a review article on the topic as reasonably can be done, given the limited space allotted.

Let the discussion begin! As a highly experienced, scientific severe-storms forecaster who was not involved (none currently in my unit were, to my knowledge), I’ll have a few tidbits to add too. There are so many places to go with this. I hope we do. Such discussion matters. I’ll post few early thoughts in multiple parts here.

Whither human understanding of the meteorological situation involved in a forecast? The article heavily emphasizes evolving educational requirements, and rightly so. It also alludes to the need for continuing meteorological and situational understanding on the part of forecasters (which begins, but does not end, in university education). The atmosphere offers continuing education, in its own language, throughout our careers as forecasters. It is professionally incumbent upon us to use observational data to read and understand the language of the atmosphere!

After all, regardless of how sophisticated the forecast models or post-processing are, we cannot optimally communicate a situation we do not deeply understand. Put another way, even the most eloquent and convincing communication of an unknowingly bad forecast, or incomplete understanding of the scenario, invites potentially tragic and economically costly consequences. Or as a colleague once put it, “You can spray-paint a turd gold, and it’s still a smelly turd.”

That means human diagnosis, by all available means (including hand analysis), still is important, and will remain so. Detailed, immersive situational diagnosis (subjective chart analyses, satellite imagery, radar imagery, soundings, radar-derived wind profiles, objective analyses, observational data of all available types) is absolutely foundational to every forecast scenario, but especially short-fused, multivariate, and high-impact ones, such as severe local storms and winter storms. Optimally thorough diagnosis is necessarily time-consuming and demanding of undistracted concentration in an operational setting. I’m glad the authors recognized and noted the importance of expert roles in forecast teams, and of fully attentive acuity. Institutionally, across public and private operations, we must reduce, not increase, distractions and disruptions to concentration on the forecast floor!

Given all the above, it is crucial for university and professional continuing education to maintain emphasis on meteorological situational understanding as the fundamental core of all forecasting and IDSS, and not lose sight of fundamentals in the zeal to embrace ever-more precise and prettier output. The safety of those who we serve with the forecasts is at stake!

More in forthcoming Part 2 and Part 3…

Heavy Blow to Scientific Credibility

by Leave a Comment

The entire theory (and research thereon) that Alzheimer’s disease is caused by “amyloid beta” protein plaques, which has been around since a widely publicized and seminal 2006 paper, now teeters on the brink of implosion, or at least deep and major reworking. A six-month Science magazine-supported investigation reveals “doctored” (pun not intended, but fits) medical imaging of the sufferers’ brains. Those images appeared inside that first paper, other papers by the same authors, and on down through more papers by others that used the original work and data as a foundation.

Many tens of millions of dollars of research, across 16 years of elapsed time, is at risk, thanks to irreproducibility and perhaps outright tampering. That risk includes waste of a lot of taxpayer funding. If these images contain bogus alterations (in effect, unethically altering data to fit hypotheses), as independent analysis hired by Science magazine have affirmed, this could become the greatest scientific scandal of our time, and that’s saying something.

Science (the profession), and medical science in particular, are having a tough time reputationally right now — most of it not deserved, but some of it well-earned, unfortunately. Science (the profession) doesn’t need any more body blows to credibility. The great majority of scientists are honest and ethical. I know many in meteorology and other geosciences, and even a few in medicine. Scientists absolutely should be held to utmost high ethical standards. Still, as with any profession, human nature can infest science with bad actors too — some of whom are smart enough to get away with it, at least for awhile.

Yes, science is self-correcting, but the time for that with this topic was *before* publication, during peer review — or at the very latest, independent reproduction shortly after with retractions…but not 16 years and countless research time and money later. Better late than never in exposing potential misconduct, of course — but this has a uniquely foul stench due to the broad impact. It’s not some obscure work with an audience of ten, investigating the electrical conductivity of a piece of tree bark in Tasmania. Instead this is a horrid disease that impacts millions of people, now and future, with enormous social and economic cost, and most of us know someone(s) affected.

As a (non-medical) scientist and photographer, who is also a journal editor, I care about experimental and data reproducibility, photographic & science ethics, as well as waste of taxpayers’ research funding. I’ve also lost friends and colleagues to Alzheimer’s disease. That all makes this revelation rather infuriating.

Please read the full, long-form article (linked here again) for the full story. Here is an excerpt:

If Schrag’s doubts are correct, Lesné’s findings were an elaborate mirage.

Schrag, who had not publicly revealed his role as a whistleblower until this article, avoids the word “fraud” in his critiques of Lesné’s work and the Cassava-related studies and does not claim to have proved misconduct. That would require access to original, complete, unpublished images and in some cases raw numerical data. “I focus on what we can see in the published images, and describe them as red flags, not final conclusions,” he says. “The data should speak for itself.”

A 6-month investigation by Science provided strong support for Schrag’s suspicions and raised questions about Lesné’s research. A leading independent image analyst and several top Alzheimer’s researchers—including George Perry of the University of Texas, San Antonio, and John Forsayeth of the University of California, San Francisco (UCSF)—reviewed most of Schrag’s findings at Science’s request. They concurred with his overall conclusions, which cast doubt on hundreds of images, including more than 70 in Lesné’s papers. Some look like “shockingly blatant” examples of image tampering, says Donna Wilcock, an Alzheimer’s expert at the University of Kentucky.

The authors “appeared to have composed figures by piecing together parts of photos from different experiments,” says Elisabeth Bik, a molecular biologist and well-known forensic image consultant. “The obtained experimental results might not have been the desired results, and that data might have been changed to … better fit a hypothesis.”