Weather or Not

Severe Outflow by R. Edwards

Independent Review Needed for Weather Disasters and Warning Verification

by Leave a Comment

As a 35-year national NWS forecaster and researcher in severe and tropical meteorology, I support the bipartisan National Weather Safety Board (NWSB) Act, which would provide an NTSB-like board review of weather disasters. Numerous other meteorologists and I have suggested something akin to this for many years.

While I agree with fellow retired NWS scientist Alan Gerard in his “Substack” column that it doesn’t do enough to evaluate non-weather disasters nor NWS performance, we shouldn’t let the perfect get in the way of the good. The NWSB Act would be a positive step toward learning from weather disasters in a systematic, consistent and accountable way, not the overpoliticized, self-interested, wagon-circling shams that the “NWS Service Assessments” often became.

Improving NWS verification services may need separate legislative action anyway, especially to address one of the most blatant, yet little-discussed, conflicts of interest that exist in all of federal government, and has for decades: verification of weather warnings by the very same offices that issue them (often, even the same forecasters).

This includes rampant use of estimated — as opposed to measured — wind and hail reports, with over 90% of thunderstorm-wind “verification” being estimates and, based on human wind-tunnel experiments, overestimates. [For much more on this, read a formally peer-reviewed, scientific publication that I lead-authored.] It’s a fox-guarding-henhouse verification system rife with opportunities for padding and inflation, regardless of how much actually happens (which is practically unknowable).

As someone who loves the NWS as a 35-year public servant there, and sees it as an extremely valuable service for the taxpayer (at a bit over a penny a person a day), its very integrity and scientific credibility depend on eliminating conflicts of interest, and ensuring optimal service through independent evaluation of both disaster services and warning performance.

One likely finding I already predict, with great confidence, will be that NWS technology and delivery each need vast improvement. Due in part to longstanding short-staffing and failure to encourage and enable true innovation, both NWS’ front-facing forecast suite (including web/app presence) and internal capabilities have become vastly outdated.

To wit, we should have been using four-dimensional, XYZ-plane-rotatable displays of model and radar fields by the time I retired, on touchscreens. That tech has been in the media at least 25 years, gaming industry since the late 1990s, Hollywood even earlier. Three-dimensional storm simulations have been around in research since 1978, on then-massive computers with less capacity than high-end gaming desktops and cloud computing of today, and finescale simulations of tornadoes and their parent supercells with Hollywood-quality graphic output were run a decade ago, on data far denser than modern operational weather models.

Instead, the most advanced severe-weather forecast operation in the world (SPC) still was — and remains — stuck with 30-year-old 2D display systems and having to mouse-click individual points on outlook lines (instead of using a stylus or finger to save time). The only 3D rotatable displays in the NWS are in licensed private-sector radar software (GR2 Analyst)!

We already should have had nationally deployed, phased-array, dual-polarization radar by 2011 when in reality, the old, greasy mechanical dishes (increasingly failure prone) were retrofitted with dual-pol. Both phased array and dual-pol tech, BTW, were in military and lab use before I was born. One is still not in our radars.

As for forecast presentation, technical discussions for other meteorologists still have their place, and should. Yet I agree with those who say we (NWS, of which I am a proud retiree) could do a much better job conveying threats in meaningful, animated, easy to understand ways. Yes, that should include robust app development and aggressive use of modern web capabilities—not handcuffing ourselves to the slowest common denominator of 486 processors and Netscape because somebody, somewhere, might still be using a teletype emulator. Hell, it wasn’t until just a decade ago that we quit using ALL CAPS in everything, like it was still 1967.

It’s baffling that we still have to issue watches with expiration times, instead of rolling them along open-ended with the threats, as we discussed among us (“amoeba watches”) for 20+ years. That predates the oft-tested, never implemented concept of “Threats in Motion” for warnings, which is very similar, and which also is long past technological capability for operationalizing. Funding (or lack thereof) has played a role, but in my experience, bureaucratic intransigence and a CYA culture frightened of innovation was at least as responsible.

Independent review would highlight shortcomings — both technological and situational — not to assign blame, but to spotlight areas for improvement. NWS, and the entire weather enterprise, should welcome true accontability!

Human Weather Forecasting in an Automation Era, Part 3: Garbage In, Garbage Out

by Leave a Comment

This short series (go to Part 1 or Part 2) arises from the recently published paper, “The Evolving Role of Humans in Weather Prediction and Communication“. Please read the paper first.

Objective verification of forecasts will remain hugely important, and the authors duly note that. But one factor not discussed (perhaps due to space limitations?) is the quality of the verification data. That matters…perhaps not to bureaucrats, who tend to overlook components of the verification sausage that provide context. But flawed verification datasets give you flawed verification numbers, even if the calculations are completely mathematically correct!

As someone who has analyzed and examined U.S. tornado, wind and hail data for most of my career, and published some research rooted in it, I can say two things with confidence:
1. It’s the most complete, precise and detailed data in the world, but
2. Precision is not necessarily accuracy. The data remain suffused with blobs of rottenness and grossly estimated or even completely fudged magnitudes, potentially giving misleading impressions on how “good” a forecast is.

How? Take the convective wind data, for example. More details can be found in my formally published paper on the subject, but suffice to say, it’s actually rather deeply contaminated, questionably accurate and surprisingly imprecise, and I’m amazed that it has generated as much useful research as it has. For example: trees and limbs can fall down in severe (50 kt, 58 mph by NWS definition) wind, subsevere wind, light wind, or no wind at all. Yet reports of downed trees and tree damage, when used to verify warnings, are bogused to severe numeric wind values by policy (as noted and cited in the paper). A patently unscientific and intellectually dishonest policy!

For another example, estimated winds tend to be overestimates, by a factor of about 1.25 in bulk, based on human wind-tunnel exposure (same paper). Yet four years after that research published, estimated gusts continue to be treated exactly like measured ones for verification (and now ML-informing) purposes. Why? Either estimated winds should be thrown out, or a pre-verification reduction factor applied to account for human overestimation. The secular increase in wind reports over the last few decades since the WSR-88D came online also should be normalized. That’s the far more scientifically justifiable approach than using the reports as-is, with no quality control nor temporal detrending.

For one more example, which we discussed just a little in the paper, all measured winds are treated the same, even though an increasing proportion come from non-AWOS, non-ASOS, non-mesonet instruments such as school and home weather stations. These are of questionable scientific validity in terms of proper exposure and calibration. The same can be said for storm-chaser and -spotter instrumentation, which may not be well-calibrated at a base level, and which may be either handheld at unknown height and exposure, or recording the slipstream if mounted on a vehicle.

Yet all those collectively populate the “severe” gust verification datasets also are used for training machine-learning algorithms — to the extent that actual, measured winds with scientific-grade, calibrated, verifiably properly sited instruments are a tiny minority of reports. With regard to wind reports, national outlooks, local warnings, and machine-learning training data use excess, non-severe wind data for verification, but because they all do, comparisons between them still may be useful, even if misleading.

Several of us severe-storms forecasters have noticed operationally that some ML-informed algorithms for generating calibrated wind probabilities put bull’s-eyes over CWAs and small parts of the country (mainly east) known to heavily use “trees down” to verify warnings, and that have much less actual severe thunderstorm wind (based on peer-reviewed studies of measured gusts, such as mine and this one by Bryan Smith) than the central and west. This has little to do with meteorology, and much to do with inconsistent and unscientific verification practices.

To improve the training data, the report-gathering and verification practices that inform it must improve, and/or the employers of the training data must apply objective filters. Will they?

This concludes the three-part series stimulated by Neil’s excellent paper. Great gratitude goes to Neil and his coauthors, and the handful who ever will read this far.

Maria’s Destruction of San Juan Radar

by Leave a Comment

Recall that when Hurricane Maria moved into Puerto Rico, the San Juan WSR-88D (Doppler radar for my non-weather friends) failed in the outer eyewall. Here’s what resulted:

The dome and antenna blew away to who-knows-where, and the steel framing itself appears slightly torqued. Trees all around are shredded to varying degrees, as should be expected with eyewall winds of an upper-echelon hurricane. The mowed-down trees in the foreground point uphill toward the radar, telling us the wind direction at the time they fell, and part of the source for embedded debris. The image below represents the last 0.5-degree reflectivity data acquired and transmitted before the radar was lost.

Now, thanks to a series of photos from NWS San Juan, two of which I include here, we can confirm why, and apply knowledge gained from damage surveys of other structures (and the Miami radar in Hurricane Andrew) to this weather-sensing structure. That is quite unfortunately ironic.

From available velocity data prior to failure (example below), we know that the radar — which sits high on a hill exposed to stronger flow than lower elevations receive — was hammered by sustained winds over 100 mph and gusts at that elevation near 150 mph. Those winds certainly contained leaves, sticks, and perhaps branches torn from nearby trees. The resulting debris-peppered airstream battered the dome piece by piece and gust by gust, until panels started loosening and coming apart under the combined stress of wind forces and high-velocity “sandblasting” effects.

Once the first panels started prying apart, the flow got inside and the entire dome shredded, exposing the antenna fully to forces for which it wasn’t designed. Being an airfoil, the dish tore away from the base easily and blew away. The entire process of destruction probably took only a few seconds, at most, from initial breach of the panels. To my knowledge, a WSR-88D hasn’t been struck directly by a significant (EF2+) tornado; however, given the windspeeds likely involved in this event, and the wind-resistance specifications of the radar structures (below, source link) one should expect similar results.

A speed of 60 m/s is the same as 134 mph. Of course the failure alone doesn’t prove the windspeed. We don’t know if the radar was built to those specs, nor to what extent the near-certain presence of flying debris played a role. Regardless, can the radar structures be engineered retroactively and cost-effectively against stronger flow?

Of course, the staff at San Juan is dealing with far more pressing matters personally and logistically than missing radar pictures and mechanical engineering of radomes. Still, it won’t help them in their public-warning mission to be without this crucial tool for at least some number of months, depending on how long it takes parts can be scavenged from elsewhere for reconstruction purposes, then shipped in and reconstructed, amidst all the island’s other damage.