A year-long study on how our local forecasts panned outBy Eric Fisher

BOSTON (CBS) Even before you read this article, let me make a few extra predictions. You will still make jokes about how forecasters get paid to be wrong. You will still say those guys and gals don’t know what they’re talking about. But you’ll also ask what the weather will be 10 days from now during your sister’s wedding ceremony or company fishing trip. Forecasting is funny like that. We have come to a point where the same old stigmas still exist, yet we expect nothing less than perfection days in advance from our local cloud-chasing guru of choice. It is so easy, even my phone does it! While you can’t tweet weather related rage to your phone app, the local meteorologist makes for a meaty target.

HOW I DID IT

The point of this year-long study wasn’t to produce an antidote for the haters. I was genuinely curious to know exactly how our forecasts panned out. I have always had a general gut feeling that they work out most of the time, but I never had hard numbers to back it up. So one contemplative night on Twitter I put out a request for someone to build an Excel spreadsheet that could help me track my forecasts, because I stink at Excel.

Lo and behold several people submitted options and I ended up partnering with Matt Cappucci, an aspiring meteorologist and student at M.I.T. and Harvard at the moment. Many thanks to Matt for breaking down the data and making this possible!

For a year, from March 2016 through February 2017, I entered forecasts for highs, lows, and precipitation after the 6 p.m. news. Instead of just using Boston, I wanted to get the full local flavor of our often-variable region. So three cities were chosen to forecast for – Boston, Worcester, and Hyannis. I figured this way we could get a good sample that included some elevation (Worcester), the urban heat island (Boston), and marine influence (Hyannis). It took a lot longer than it sounds to forecast 8 days out for three different cities. I’m sure a person more technologically savvy could have found a better way to do it, but I just manually pecked away at the keyboard inputting all the data each night.

When viewing some of the results, keep in mind that we end every newscast with a 7-day forecast. Day 2 represents what is on deck for the following day, while Day 8 is a full week in advance.

7day How Accurate Are WBZ Weather Forecasts? A Case Study

Without further ado, let’s dig in!

TEMPERATURES

One thing that will come as no surprise is that forecasts become more difficult over time. This is because weather operates on chaos theory. We start with the best possible initial conditions, but there are inevitably some errors at that starting point. The farther out you go, the more those compound and end up heading down unexpected paths. The vast majority of the time your best temperature forecast is going to be for the next day or two; you will be off by a larger amount the farther into the future you go.

dailyhightemps How Accurate Are WBZ Weather Forecasts? A Case Study

When it came to high temperatures, the error for the first day of the forecast was 2.19F and grew to 4.74F by the final day of the 7-Day. I was very happy to see this! Every part of the country may have a different definition of a ‘good’ forecast because some climates are more variable than others (if you’re missing a forecast by 5 degrees in Florida, you’re doing something wrong).

I have always thought that you want to be within 3 degrees for a perfect forecast a day or two in advance. Farther out, you do not want to be missing by more than 5 degrees. So these results fit nicely into expectations! The short term outlooks averaged out to < 3F and the longer term outlooks stayed < 5F. Knowing that you can expect a day a week in advance to be within 5 degrees of the forecast is pretty solid for planning purposes.

The low temperature departures from actual values fell from 4.61 degrees to 2.44 as each week wore on.

lowtempdays How Accurate Are WBZ Weather Forecasts? A Case Study

Oddly, my best temperature forecasts came in the wintertime. One would think that winter and spring are the most difficult seasons to forecast for in New England, so that was a bit of a surprise. That being said, spring lived up to its reputation. It is a constant battle trying to figure out when a sea breeze will kick in or if it will relax at the end of the day and allow for a last minute 10 degree jump! Boston is particularly hard to predict in this regard. I also feel like I was just starting to get the hang of the forecasting ‘game’ for this experiment in spring, and was sometimes forgetting to think about midnight highs/lows. You generally are trying to get across the idea of what the day and night will be like, and can forget that for a ‘point’ forecast you have to pick the right number regardless of the time of day it occurs.

seasonalhigh How Accurate Are WBZ Weather Forecasts? A Case Study

So which city is toughest? It looks to be Hyannis, which was a shock to me. I thought for sure that Boston would be the hardest point and Worcester the easiest. That’s because having the observations at Logan is challenging (on the water, subject to be an outlier vs. just a mile or two inland), and Worcester has no ocean influence to factor in most of the time. I found overnight lows to be the most humbling part of Cape Cod/Islands forecasting, and was also sometimes surprised to see how long it took for sea breezes to halt the morning temperature rise in the spring/summer (the sensor is a mile inland at Barnstable Municipal Airport).

temperrors How Accurate Are WBZ Weather Forecasts? A Case Study

PRECIPITATION

We get plenty of water whether by raindrop, snowflake, or sleet pellet in southern New England, but the astute weather observer would recall that we had a drought during the time of this experiment. So that’s easy, right? Wrong. Most of our rain events were very small ones – a few hundredths here, a tenth or two there. That is a lot harder to get right than clear-cut bigger weather systems bringing widespread rainfall. It was a real pain.

Eight days out, the average error was only 0.29” off when it came to rainfall. Strangely, the best precipitation forecast came four days in advance, and then wobbled upwards in terms of error. There was an average error of 0.12” on Day 4, but the error one day out rivaled the error five days in advance. Kind of wonky results here. Forecasting has a mental aspect, and sometimes I think it is easier to forecast several days out when you are going on feel, as opposed to trying to get it exactly right and overthinking things a day out. Maybe that’s the reason rain/snow forecasts were better four days in advance than the night before?

How about a less technical question. When I said it would rain, did it? To answer this question, we neglected specific amounts, and simply went with a Y/N whether or not the drops fell. Let’s consider Boston. A week out, the accuracy rating was 73% in calling a hit or a miss; the day before an event, 89%. I was very happy to see the predictions were right 9 out of 10 times to help people plan their days! Three out of four a week in advance allows for some informed decision-making as well, which is really what a longer range outlook is all about – trying to make some generally well informed decisions about what to do in advance.

precip1 How Accurate Are WBZ Weather Forecasts? A Case Study

THE TAKEAWAY

I wanted to start this project because I never really had a solid answer for “how often are your predictions right?” I had a good idea that they weren’t terrible, and thought they generally ended up doing well. Now there is some evidence to show performance. Will we still blow a forecast every once in a while? You betcha. Prediction in any career is not easy; just ask pollsters and stock analysts. Even Vegas loses every once in a while. But as this undertaking shows, you can take a forecast to the bank the vast majority of the time.

This also brings about a problem. People expect pinpoint accuracy. In this day and age, being remotely wrong is reason for haters to go on the attack. Comes with the territory. If a snow forecast is off by 3 inches, people want you publicly stoned. However, that is a testament to the vast improvements in weather forecasting over the decades. We can only hope that it will continue to improve in the years to come, making emergency management and your local round of golf that much easier to plan!

Comments (5)
  1. Art Winters says:

    This has little or nothing to do with “hating”. Forecasts should not be made more than 3 days out. Also, go back further, to 2-3 years. I would be willing to bet that 6-7 days out, one could flip a coin and do just as well. Also, admit that you are wrong!! Rarely do I hear that from our local forecasters. There is a level of arrogance that seems to be built in. Maybe that’s where the perception of hate comes in.

  2. Eric, this is cool. If you’d be willing to share your data I’d love to dig in! I bet a lot of folks on kaggle.com would also find it interesting.

  3. Great article! This summer if felt like the temp forecasts were consistently too low. Does the data bear this out?

Leave a Reply

Please log in using one of these methods to post your comment:

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s