Monthly Archives: January 2015

Forecasting uncertainty in practice: Snowperbole

Example of snow forecast communicating levels of undertainty; image from the Capital Weather Gang

Example of snow forecast communicating levels of uncertainty; image from the Capital Weather Gang

Because making accurate predictions is extremely difficult, we can dramatically improve both the accuracy of forecasts and enable effective communication about the forecast by embracing the uncertainty involved in the forecast. This allows decision-makers to both use the information available while understanding the limits of those predictions.

Following forecasts for a “potentially historic” storm set to hit New York and New England, public officials in New York City went to great lengths to emphasize the dangers of the storm. The Governor closed down New York’s subways in anticipation of the storm (showing one of the quirks of New York’s transit governance, local transit is under state control).

There was just one problem: the storm mostly missed NYC.

In their forecast post-mortem, the Washington Post’s Capital Weather Gang highlighted the key shortcomings of the forecast – a failure to present the level of uncertainty in the forecast.

Why were the forecasts so bad?

It’s simple: Many forecasters failed to adequately communicate the uncertainty in what was an extremely complicated forecast. Instead of presenting the forecast as a range of possibilities, many outlets simply presented the worst-case scenario.

Especially for New York City, some computer model forecasts were extremely dire, predicting upwards of 30 inches of snow – shattering all-time snowfall records. The models producing these forecasts (the NAM model and European model) had a sufficiently good enough track record to take them seriously.

However, some model forecasts (e.g. the GFS model) signaled reason for caution. They predicted closer to a foot of snow.

Part of the challenge here is that most of the forecast was accurate. This was a historic storm; the storm simply tracked a bit further to the east. Areas like New York City were right on the margins, where a small change to the inputs can mean a large change in the outcome  – and the forecast did not adequately convey that uncertainty. Add in the fact that the forecast miss happened to be the largest city in the United States, and you have a very public error.

When a forecast is so sensitive to small changes (eastern Long Island, not far away, received 30-plus inches), it is imperative to loudly convey the reality that small changes could have profound effects on what actually happens.

It’s easy to second-guess public officials making key decisions like closing transit systems after the fact (and after the forecast bust), but they can only act on the information that they have in front of them. It’s easy to argue that it is better to be safe than sorry (and this is certainly true – it is better safe than sorry) but there is a real risk of eroding public confidence in these kinds of decisions when the forecast doesn’t pan out. (It doesn’t help that despite closing the subways, the MTA’s snow plan called for trains to remain in operation without passengers to keep the tracks clear of snow)

As some meteorologists suggest, conveying the uncertainty in their forecasts should be a larger element of both the forecast and communication. It’s not just a matter of using the best information available, but also understanding the uncertainty involved.

The cone of uncertainty

One of the elements that makes prediction difficult is uncertainty. In one of the chapters of Donald Shoup’s High Cost of Free Parking (adapted for Access here), Professor Shoup poses the question:

HOW FAR IS IT from San Diego to San Francisco? An estimate of 632.125 miles is precise—but not accurate. An estimate of somewhere between 400 and 500 miles is less precise but more accurate because the correct answer is 460 miles. Nevertheless, if you had no idea how far it is from San Diego to San Francisco, whom would you believe: someone who confidently says 632.125 miles, or someone who tentatively says somewhere between 400 and 500 miles? Probably the first, because precision implies certainty.

Shoup uses this example to illustrate the illusion of certainty present in the parking and trip generation estimates from the Institute of Transportation Engineers. Many of the rates are based on small samples of potentially unrepresentative cases – often with a very wide range of observed parking/trip generation. Shoup’s concluding paragraph states:

Placing unwarranted trust in the accuracy of these precise but uncertain data leads to bad policy choices. Being roughly right is better than being precisely wrong. We need less precision—and more truth—in transportation planning

Part of the challenge is not just knowing the limitations of the data, but also understanding the ultimate goals for policy. David Levinson notes that most municipalities simply adopt these rates as requirements for off-street parking. This translation of parking estimates to hard-and-fast regulation is “odd” in and of itself. What is the purpose of a parking requirement? To meet the demand generated by new development?

Parking demand for a given building will be a range throughout the course of a day and a year, and demand for any given building category will itself fall within a large range. That range is reality, but that unfortunately doesn’t translate into simply codified regulations.

In the previous post, I discussed the challenges of accurate prediction and specifically referenced Nate Silver’s work on documenting the many failures and few successes in accurate forecasting. One area where forecasting improved tremendously is in meteorology – weather forecasts have been steadily improving – and a large part of that is disclosing the uncertainty involved in the forecasts. One example is in hurricane forecasts, where instead of publicizing just the predicted hurricane track, they also show the ‘cone of uncertainty‘ where the hurricane might end up:

Example of a hurricane forecast with the cone of uncertainty - image from NOAA.

Example of a hurricane forecast with the cone of uncertainty – image from NOAA.

So, why not apply these methods to city planning? A few ideas: as hypothesized before, the primary goal for parking regulations isn’t to develop the most accurate forecasts. The incentives for weather forecasting are different. The shifts to embrace uncertainty stems from a desire finding the most effective way to communicate the forecast to the population. There are a whole host of forecast models that can predict a hurricane track, but their individual results can be a bit messy – producing a ‘spaghetti plot,’ often with divergent results. The cone of uncertainty both embraces the lack of precision in the forecast, but also simplifies communication.

For zoning, a hard and fast requirement doesn’t lend itself to any cone of uncertainty. Expressing demand in terms of a plausible range means that the actual requirement would need to be set at the low end of that range – and in urban examples, the low end of potential parking demand for any given project could be zero. Of course, unlike weather forecasts, these regulations and policies are political creations, not scientific predictions.

Meteorologists also have the benefit of immediate feedback. We will know how well hurricane forecasters did within a matter of days, and even then we will have the benefit of several days of iterations to better hone that forecast. Comparatively, many cities added on-site parking requirements to their zoning codes in the 1960s; regulations that often persist today. Donald Shoup didn’t publish his parking opus until 2005.

There’s also the matter of influencing one’s environment. Another key difference between a hurricane forecast and zoning codes is that the weather forecasters are looking to predict natural phenomena; ITE is trying to predict human behavior – and the very requirements cities impose based on those predictions will themselves influence human behavior. Build unnecessary parking spaces, and eventually those spaces will find a use – inducing the very demand they were built to satisfy. There, the impacts of ignoring uncertainty can be long-lasting.

Here’s to embracing the cone of uncertainty!

Prediction is hard – so why do we make key decisions based on bad information?

Comparison of USDOT predictions for Vehicle Miles Traveled, compared to actual values. Chart from SSTI.

Comparison of USDOT predictions for Vehicle Miles Traveled, compared to actual values. Chart from SSTI.

Back in December, David Levinson put up a wonderful post with graphical representations looking to match predictions to reality. The results aren’t good for the predictors. Lots of official forecasts call for increased vehicle travel, while many places have seen stagnant or declining VMT. It’s not just a problem for traffic engineers, but for a variety of professions (I took note of similar challenges for airport traffic here previously).

Prediction is hard. What’s curious for cities is that despite the inherent challenges of developing an accurate forecast, we nonetheless bet the house on those numbers with expensive regulations (e.g. requiring off-street parking to meet demand) and projects (building more road capacity to relieve congestion) based on bad information and incorrect assumptions.

One of the books I’ve included in the reading list is Nate Silver’s The Signal and the Noise, Silver’s discussion of why most efforts at prediction fail. In Matt Yglesias’s review of the book, he summarizes Silver’s core argument: “For all that modern technology has enhanced our computational abilities, there are still an awful lot of ways for predictions to go wrong thanks to bad incentives and bad methods.”

Silver rose to prominence by successfully forecasting US elections based on available polling data. In the process, he argued the spin of pundits added nothing to the discussion; political analysts were seldom held accountable for their bad analysis. Yet, because of the incentives for punditry, these analysts with poor track records continued to get work and airtime.

Traffic forecasts have a lot in common with political punditry – many of the projects are woefully incorrect; the methods for predicting are based more on ideology than observation and analysis.

More troubling, for city planning, is the tendency to take these kinds of projections and enshrine them in our regulations, such as the way that the ITE (Institute of Transportation Engineers) projections for parking demand are translated into zoning code requirements for on-site parking. Levinson again:

But this requirement itself is odd, and leads to the construction of excess off-street parking, since at least some of that parking is vacant 300, 350, 360, or even 364 days per year depending on how tight you set the threshold and how flat the peak demand is seasonally. Is it really worth vacant paved impervious surface 364 days so that 1 day there is no spillover to nearby streets?

In other words, the ideology behind the requirement wants to maximize parking.

It’s not just the ideology behind these projections that is suspect; the methods are also questionable at best. In the fall 2014 issue of Access, Adam Millard-Ball discusses the methodological flaws of ITE’s parking generation estimates. (Streetsblog has a summary available) Millard-Ball notes that the “seemingly mundane” work of traffic analysis has enormous consequences for the shape of our built environment, due to the associated requirements for new development. Indeed, the trip generation estimates for any given project appear to massively overestimate the actual impact on traffic.

There are three big problems with the ITE estimates: first, they massively overestimate the actual traffic generated by a new development, due to non-representative samples and small sample sizes. Second, the estimates confuse marginal and average trip generation. Build a replacement court house, Millard-Bell notes, and you won’t generate new trips to the court – you’ll just move them. Third, the rates have a big issue with scale. Are we concerned about the trips generated to determine the impact on a local street, or on a neighborhood, or the city, or the region?

What is clear is that these estimates aren’t accurate. Why do we continue to use them as the basis of important policy decisions? Why continue to make decisions based on bad information? A few hypotheses:

  • Path dependence and sticky regulations: Once these kinds of regulations and procedures are in place, they are hard to change. Altering parking requirements in a zoning code can seem simple, but could take a long time. In DC, the 2006 Comprehensive Plan recommended a review and re-write of the zoning code. That process started in earnest in 2007. Final action didn’t come until late in 2014, with implementation still to come – and even then, only after some serious alterations of the initial proposals.
  • Leverage: Even if everyone knows these estimates are garbage, the forecasts of large traffic impacts provide useful leverage for cities and citizens to leverage improvements and other contributions from developers. As Let’s Go LA notes, “traffic forecasting works that way because politicians want it to work that way.”
  • Rent seeking: There’s money to be made from consultants and others in developing these inaccurate estimates and then proposing remedies to them.