Systematic Private Sector Forecast Verification?

For decades, a few of the largest domestic, private weather forecasting operations have claimed they outforecast your National Weather Service, using seemingly spectacular examples in a firestorm of press releases and advertisement. I won’t dignify these outfits by naming them; though my last blog did deal with one, and the other principal offender is based in Wichita.

It’s a brash, clever game on the part of those particular private outfits’ PR machines, in part because it takes full advantage of:

    1. The governmental forecasting agency’s utter ineptitude at corporate PR,
    2. Its morbid fear of offending those same attack dogs in the private sector, and
    3. The handcuffing by their management of field forecasters’ ability to publicly comment on their successes, in comparison to and with the same unbridled force as the private sector (level playing field!).

Hey, I didn’t say those private sector weather executives from State College and Wichita were dumb. Arrogant and insidiously manipulative, yes…but not stupid! The myth behind their forecasting “superiority” lies in how they advertise their so-called success. Instead of systematic, scientifically robust verification statistics across a wide range of their forecasts, as the NWS at least partly provides, they cite selectively hand-picked examples, embellished with gooey gumdrops of self-promotional hyperbolae, absolutely omitting any and all failures.

Merely claiming forecast success is laughably easy. If I am a private forecasting manager, and I carefully choose:

    1. My “best” forecasts
    2. The criteria by which I want to measure or chart them to make them look “best”
    3. Which NWS products appear to compare most unfavorably to mine

….then I’m going to look like the ultimate superstar of forecasting before any audience that wouldn’t know better. That audience includes almost all current and potential professional clients, and the overwhelming majority of the public at large.

That’s what the Wichita and Pennsylvania boys are best at — not the forecasting itself necessarily, but the selective self promotion of forecast “successes”, using smoke and mirrors and occasionally a valid statistic or two to appear smart.

Those dudes don’t give a flip about scientifically meaningful, systematic verification; and it doesn’t seem to matter to 99.9% of clients as long as you can make them think you did well for select events. In the corporate world it’s about impressions, not reality. So hard science just isn’t done in terms of verifying larege numbers of private sector forecasts.

It’s a devious form of manipulation-by-omission, but one over which government forecasters have negligible
influence thanks to numerous systematic constraints inherent to who they work for. [For a classic example of such constraints, a memo was sent out earlier this year to one region’s forecasters directing them that, “You must state that you have no opinion” regarding the Texas meteorological certification issue. Other examples of internally imposed censorship abound in the long annals of public sector meteorology, dating back to Cleveland Abbe and Isaac Cline.]

It’s hard to defend oneself while fastened securely in a straitjacket.

It doesn’t help the government forecasters either that, by the very nature of predicting the future in a fluid (air) dominated by chaos, occasional forecasts miss. Then someone, somewhere, may lay valid claim that they did it better in that particular event, simply because there are so many people forecasting any given snowstorm or hurricane. [The storm chaser analog is that someone always will get “better” video than you because there are hundreds and hundreds of chasers on any given butt-kicking, tornadic supercell.]

There’s nothing that can be done directly about such deceit and tomfoolery. Deceptive self-promotion predates snake-oil salesmen and is an inevitable consequence of profit-driven competition. As a Republican and capitalist, I favor free enterprise and want private forecasting companies to succeed — just not at the expense of those who are paid by the paxpayers to warn everyone, instead of just warning lucrative clients.

The generally naive masses in Germany in the late 30s started believing Hitler, who (though evil as can be) was a gifted persuasive speaker, because they had no well conceived counter-message available. By contrast I don’t believe those most flamboyant of the private weather executives are “evil” at all…just misguided, arrogant and suffering from delusions of inflated ability. The propaganda philosophy is much the same as with dictatorships (Nazi) and oppressive theocracies (Taliban) where the alternative presentation is systematically underrepresented or handicapped, or where the folks with the best understanding of their inabilities don’t have the access needed to expose them. NWS, though, sorely needs better PR for the events it does nail, because the disproportionate sound of meteorological trumpets is coming from a few very loud private sector bosses.

These guys have been whining for years that NWS needs to stop trying to compete with them. Even if one accepts the dubious and unproven idea that it’s true, then the cease-fire still needs to work both ways! In principle, if the private weather executives in Wichita and State College don’t want NWS trying to compete with them, then the kettle should also stop calling the pot black by propagating “We beat the Hurricane Center” type propaganda. It’s classic hypocrisy…do as we say, not as we do.

The American Meteorological Society could lean on private companies harder to stop playing competition games, under the noble premise that forecasting for severe/extreme events (i.e. hurricanes) should be *cooperative* and not *competitive* between NWS and the private sector. Darn right, it should. Human lives are at stake, and that trumps profit! Oh, but will the AMS take that risk? Do dogs meow? AMS gets tons of cash (here we go again, talking about AMS and money) from the private sector and would not risk losing it by having the courage to go that far out on a limb.

As a former member of the AMS Severe Local Storms Committee, at the end of the process, I was only peripherally involved in crafting the AMS Tornado Forecasting and Warning Statement. Criticisms from people in those very outfits in the review process led to dilution of original language that explicitly and strongly established that the NWS needs to remain the sole source of all tornado warnings. Unfortunately, I was in the minority on taking and holding a strong stand about the sole voice of tornado prediction remaining in NWS, and not giving an inch to any erosional foolishness from a very tiny and shrill minority of egomaniacs in the private sector.

I won’t get into detail about the spectacular flameouts of private sector forecasts — such as the enormously costly and unnecessary forecast from the State College outfit that Hurricane Gilbert would hit Galveston in 1988. Instead, I challenge those companies to provide verification of *all* their temperature, wind, rain/snow and hurricane track/intensity forecasts for public and scientific review, as NWS does! Dogs will meow when that happens also, because the grand facade will be exposed the the house of cards will fly apart as if dropped into the eyewall of Gilbert.

So, in the end, the government forecasters are rather stuck in a lose-lose game. The best that NWS meteorologists can do is sidestep the incoming fire whenever possible and keep issuing the consistently superior forecasts, hoping they will stand on their own merit before enough eyes to matter. Perhaps, toss out a dollop of (accurate!) self promotion of NWS successes for PR sake wherever one can get away with it.

Just my personal $.01 (the other $.01 went to yet more outrageous AMS page charges and conference registration fees)…


2 Responses to “Systematic Private Sector Forecast Verification?”

  1. Helping Hand on April 29th, 2005 4:00 am

    While NWS meteorologists cannot lobby Congress, there is nothing stopping eligible Penn State Alumni from voting against Joel Myers in the current election for Alumni Trustees. I am sure there are many in the NWS. This would undercut one of his power bases.

    Time is short, the ballots are counted by May 12 this year. However, there is electronic voting. The contact information is for the trustees is:

    Board of Trustees
    The Pennsylvania State University
    205 Old Main
    University Park, PA 16802-1571
    Office: (814) 865-2521
    Fax: (814) 863-4631
    Paula R. Ammerman
    Director, Office of the Board of Trustees/
    Associate Secretary

    Below is the section on Alumni trustees from the PSU Board of Trustees web site.

    Alumni trustees are elected, three each year for three-year terms, by the following procedure: On or before January 15 each year, nominating ballots are sent (1) to all alumni of the University who have, within two years prior to March 1 of each year, been either active members of the Penn State Alumni Association or contributors to the Penn State Fund, or (2) to any other alumni who make a request in writing for a ballot.

    Qualified alumni who receive 50 or more nominating votes, and who accept the nomination in writing, have their names placed on an election ballot which is sent to the alumni (specified in the paragraph above) on or before April 10. At a time specified by the Board during spring commencement week, the ballots are tabulated in the presence of two trustee tellers. The three candidates receiving the highest number of votes are declared elected

  2. Steve G. on May 1st, 2005 5:31 am


    Good read. And “Helping Hand,” thanks for the “helping hand.” I’ll do my duty! : )


Leave a Reply

You must be logged in to post a comment.