Data-Powered Forecasting
The results of the recent presidential election in the USA came as a surpise to some. Most of the pundits and forecasters on the Republican side had predicted a victory for their candidate, while those on the Democratic side were quite certain theirs would win. A “Pundarts” graphic at the Slate.com website illustrates the relative success of these two groups in predicting the final outcome.
Right in the bullseye was Nate Silver, blogger for the New York Times, now hailed as the “golden boy of electoral statistics.” He was spot-on this year, as he was four years ago. No wonder that sales of his book The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t jumped some 850% the day after the election.
According to Silver, in spite of how important forecasting is to our daily lives, we are remarkably poor at it. Those who are most successful tend to be more modest, less ideological, and to rely on empirical evidence. The key is to be able to successfully sift through a sea of noise to detect the signal. Much of that noise can be internal, coming from the pundits themselves in the form of personality traits, subconsious biases, and pride in being an expert.
While we don’t have much control over the human nature of forecasters, real-time cloud computing opens new possibilities for gathering empirical evidence. Those currently using real-time systems already understand the value of real-time data. Connecting to the cloud provides more depth and reach for these systems, as hard data and empirical evidence become more readily available, and up-to-the-second. The growing availability of data, broad-based and timely, is moving us out of the realm of supposition into higher levels of certainty.
Take Nate Silver’s own success, for example. He didn’t conjure up an imaginary all-knowing genie, or shake a Magic 8 Ball. He certainly didn’t try to advance his own “expert” opinion. He simply looked at the average of a number of polls. He worked with the hard facts, considered historical precedence, and made reasonable guesses with a stated level of probability. In addition to these, he was willing and able to adapt to rapidly changing conditions.
If you read his blog entries and other reports, you will see that as the polls changed over time, Silver changed his predictions as quickly as possible. He was recalculating up until the day of the election, and even during the hours that the returns were coming in he was posting updates to his blog.
All successful forecasting, be it for the weather, the stock market, or business planning, ultimately relies on hard data. In our ever-accelerating world we are becoming increasingly aware of the need for timeliness of that data. The more recent the data, the better the forecast. And in our growing interconnectedness, we are discovering the value of having full access to that data anywhere, any time.
The availability and speed of the incoming data are constantly increasing. Where will all of this end up? Will the past, present, and future eventually get compressed into real time, making data spontaneously available everywhere? Are we ready to consider the possibility of going beyond the guesswork of forecasting, to realize a new reality, the certainty of now?
Not quite yet. We’ll take up that topic soon.
