https://amgreatness.com/2017/02/07/nate-silver-fetish-data-driven-journalism/
With the UK election on top of us, once again things are totally confused, both main political parties are in disarray, the minor parties have once again blown their chances of any significant breakthrough by virtue of sheer ineptitude and the big data boys with their graphs, charts and tables, are running around like headless chickens because opinion polls are pointing in very different directions.
In 2012 some little statistics nerd named Nate Silver was elevated to the pantheon of Technology Gods, when he correctly predicted, having modelled the outcome of the vote on his meta - analysis of opinion polls, that Barack Obama would win a second term as US President. That was the election in which Obama's opponent Mitt Romney famously threw the fat lady off the stage long before she had even done her warm up scales, when he conceded a key state with less than twenty five percent of the votes counted.
It was the comedian Grouch Marx who said "The show ain't over til the fat lady sings" in a 1930s comedy. More recently somebody else said, "It ain’t over till it’s over." With the advent of Big Data and reliance upon data-driven modeling of climates, national economies, political elections, economic performance and sports events we might just as well rephrase it as, “It’s over before it begins.” Which is exactly what one TV pundit did after that Romney concession. When his colleague commented that it seemed a little early to concede, the man, whose name I don't know and have no interest in rediscovering, said something like, "Come on, you know how the science works." But did we?
Obviously not, given what happened in Britain's in / out referendum on continued EU membership and the US election of Donald Trump to the presidency in 2016. On the day before the election the divine wisdom of Nate Silver's digital models gave both the Keep Britain In the EU campaign and Hillary Clinton better than 80% chances of winning.
Another spectacular Nate Silver cock up in 2016 related to American Football's top prize Super Bowl, Nate Silver’s FiveThirtyEight.com website predicted the New England Patriots to win. Going into the half-time with Atlanta Falcons leading 28-3, the site gave the Patriots a less than 1 percent chance of winning. FiveThirtyEight tweeted: “That Patriots drive took another 5:07 off the clock and actually dropped their win probability from 1.1% to 0.5%.”
Of course we all know what happened next. In yet another brilliant statistical upset that exposed Nate Silver's infallible mathematical modelling as no better than guesswork, in the second half the Patriots turned mathematical modelling on its head. They scored 25 unanswered points to win, giving America an timely lesson in how self - belief, determination, and the ability of human beings to make mockery of statistics.
It was a lesson for those who tell us we must believe in science and mathematics that sometimes the whole is greater than the sum of the parts, a reminder of the value of commitment, fortitude and preparedness to risk everything; values that were once an integral part of the American mythos, and drove the European pioneers who founded the American nation, but that increasingly become unfashionable and been sidelined in exchange for the perceived infallibility of data-driven analysis and computer modelling.
Since information (data) replaced wisdom and judgement and became, for the media at least, the highest level of human intellect, as demonstrated by Nate Silver’s elevation to divine status and FiveThirtyEight’s place in pop culture, a cultural shift has taken place with regard to the use of statistics. No longer were the words of nineteenth century statesman George Canning, "I can make statistics tell me anything except the truth," quoted at those who relied on Big Data, polling, and more mathematical models. On TV shows and social media feeds the pronouncements of nerds with tablet computers were treated as if they were inscribed on tablets of stone. Data-driven analysis, whether accurate or not,(and it was not accurate as often as it was) provided a quick and unchallengable way to excoriate opponents in debate who relied on old fashioned things like empirical evidence. “Silver gives Hilary a x++ percent chance to win the election” became the Trump card in any debate on the US election. Nate Silver gives the Remain campaign x+ chance of winning the Brexit referendum was offered to counter any mention of the anti - E.U. mood in the old English industrial areas of the north and midlands, in South Wales and central Scotland.
And in both cases the fat lady was singing at the top of her range, but only those with the ability to see how the mathematical models were bring manipulated to produce politically expedient answers could see the abyss opening up under the predictions of the big data boys.
We had reached a point where pundits were willing to assign data modelling more value than observable reality. The irrationality and risk-taking inherent in human decision-making were brushed aside, this was the age of nerds. In politics, sport and weather forecasting, statistical models were held up as unassailable predictors.
And in all cases, in 2016, they were wrong. Trump and Brexit, New England Patriots in the US Super Bowl and Leicester City in the English Premier League soccer, and as for climate change predictions, well snow should have been unknown to European and American children by now, the northern ice cap should have disappeared and we were promised 50 million refugees, displaced by rapidly rising sea levels would be wandering the world looking for a new home (that last one was particularly stupid because if all the predictions had come true those people could have settled in the sub - tropical paradises of Siberia, Alaska and northern Canada.)
For his part, no matter how certain Silver's fans might have been of the infallibility of his model, the man himself, more often than not, would hedge when asked a direct question. Like those climate scientists whose opinions trigger predictions of sure and certain destruction of life on earth by half past two tomorrow if we do not abandon coal and oil NOW!, Nate's own predictions are fenced in qualifiers, possible, probably, is likely to etc. In October 2016, under a headline that read “Clinton Probably Finished Off Trump Last Night,” Silver commented: “I’m not sure I need to tell you this, but Hillary Clinton is probably going to be the next president. It’s just a question of what ‘probably’ means.” He then spent the bulk of the article presenting the statistical case to back up his claim that Clinton would win, but in the end equivocated again, acknowledging that he could be mistaken, that other signs pointed to the possibility of another outcome.
Earlier in the election cycle, when results of the Republican primary, the Michigan Democratic primary, and the general election proved him very wrong, Silver’s excuses involved not moving the goalposts so much as moving the entire effing stadium. He claimed that certain unprecedented events had influenced voters (Wikileaks publishing documents from the DNC 'hack'? The Democrat candidate being a known liar and abuser of office for personal gain?) which skewed the initial models. Even after the Super Bowl, in an attempt to make light of the situation, he tweeted: “At least the Falcons won the popular vote.” To which a user responded, “Nate, you don’t get to make election jokes.”
Silver also acknowledged that in any prediction based on mathematical modelling of data, subjective best guesses and assumptions must necessarily be factored into the algorithm. When unprecedented things happen those guesses and assumptions cannot easily be factored out. By saying this Silver admits that statistical analysis of data might work for p lotting trends but cannot predict outcomes when people are involved and instinct, emotion and individual circumstances come into play. This was best summed up in an article by David Morris, on Silver’s failure to predict Trump’s victory in the Republican primary: “Unlikely events like the Trump nomination are, by their very nature, impossible to predict.”
The models are not ancient oracles, they do not predict the future. At best they are informed guesses on the outcome of events conformed to the established patters of similar events in the past. The trouble with trusting the Oracle, however, is that when events occur, though there may be a superficial similarity, there at many different factors at play.
Nate Silver’s infallibility is therefore not the issue. Everyone get things wrong. Silver made his money and his name as a poker player, not one of those pallid, intense characters who spent their lives in smoky, poorly lit back rooms, that we see in movies, but a player against computers in online games. This is interesting because against a computer it's possible to play the percentages and win. In those smoke filled, dimply lit back rooms, emotions, stresses. personal qualities and other things that cannot be factored into a computer program are at work. And then there are the bluffs and the psychology.
Yet despite his being fabulously wrong many times, and despite his admissions of fallibility, people still cling to his as the ultimate argument from authority. "Nate Silver says," you will hear as confusion reigns ahead of the UK election, with polls showing leads for the conservatives ranging between one and twelve percent (i.e. a hung parliament to a super - majority). This signals a problem with the scientific, control freak culture leaders such as Barack Obama, Tony Blair, David Cameron and Angela Merkel have been eager to impose. Science demands simple answers and seeks to provide such answers to complex and often unquantifiable questions. When those questions relate to a globalist, scientific views of how the world should be, we should be doubly diligent in questioning 'the science'.
It’s not Silver. He’s just a poster - boy for the Church of Sciencology Cult. The irony is that mathematics is not really a science, just as Julius Caesar was proclaimed a god by the mob, so mathematics has been proclaimed a science by an academic mob. In fact it is, in the truest sense of the word, an art; an artifice, something created by humans. Nature does no mathematics, a couple of years ago in a BBC Science documentary on British TV two mathematicians were burbling ecstatically about how the invention of 'zero' (by some ancient Assyrian, Babylonian, Egyptian or Hindu philosopher or had changed our understanding of the universe. Well it may have changed the way we do calculations but it does not change the fact that there is no zero in nature. There may be an absence of something (breathable air for example) but there can never be zero nitrogen, oxygen, carbon dioxide atoms to nature, that would only be significant to a human trying to measure what was (or wasn't) present.
Jason Rhode, in Paste Magazine, wrote a withering critique of Silver, opening with a quotation from Federalist 55: “Nothing can be more fallacious than to found our political calculations on arithmetical principles.” And yet today many seem to believe that Silver is The God Mathematics made flesh, an avatar of a desire for certainty in the unpredictable and unscientific world of human interactions. As novelist Terry Pratchett wrote in "The Thief Of Time," in any battle between order and chaos, chaos will win because it is better organised.
Invoking FiveThirtyEight seemed, before Brexit and Trump, to bestow upon those who quoted the Silverian doctrines an air of both intellectual superiority and mathematical indifference. “Nate Silver predicts…” is akin to saying “Shut up idiot, what do you know? If you don't know 'the science works' you are not entitled to an opinion. The numbers can’t lie, because science!” But that appeal to Silver is really an appeal to the desire of insecure people for a totally stable and predictable future. So how will it play out in Britain? Here are the headline figures for the most recent polls:
Key: Con - Conservative; Lab = Labour; LDem = Liberal Democrat; UKIP = United Kingdom Independence Party
June 5:
ICM / Guardian. CON 45%(nc), LAB 34%(nc), LDEM 8%(-1), UKIP 5%(nc). Fieldwork was Friday to Sunday, so will have been largely before the terrorist attack in London Bridge.
June 4:
Opinium / Observer. CON 43%(-2), LAB 37%(+2), LDEM 6%(-1), UKIP 5%(nc). Fieldwork was on Tuesday and Wednesday and changes are from a week ago.
Com Res / Sunday Mirror. CON 47%(+1), LAB 35(+1), LDEM 8%(nc), UKIP 4%(nc). Fieldwork was Wednesday to Friday and changes are from a week ago.
ICM / Sun. CON 45%(nc), LAB 34%(+1), LDEM 9%(+1), UKIP 5%(nc). Changes are from the ICM poll for the Guardian at the start of the week.
You Gov / Sunday Times. CON 42%, LAB 38%, LDEM 9%, UKIP 4%. Fieldwork was Thursday to Friday
Orb / Sunday Telegraph. CON 45%(+1), LAB 36%(-2), LDEM 8%(+1), UKIP 4%(-1). Fieldwork was Wed-Thurs with changes from a week ago.
Survation / Mail on Sunday. CON 40%(-6), LAB 39%(+5), LDEM 8%(nc), UKIP 5%(+2). Fieldwork was conducted wholly this morning (after London Bridge incident).
June 2:
IPSOS Mori / Evening standard. CON 45%(-4), LAB 40%(+6), LDEM 7%(nc). Changes are from their previous poll in mid-May
Source: UK Polling report
Discounting the one percent Conservative lead in the Survation / Mail on Sunday poll as it was an instant raction type of thing, and also because no sample details were made available, we see widely differing results but all pointing to a win for the conservatives. In the past however, while a meta - analysis or poll of polls has been reasonably accurate in predicting overall percentages of the vote for different parties, those percentages cannot project the number of seats won in Parliament, because of the way our system works. Similarly, in the USA in 2016, while Hillary Clinton won the popular vote by virtue of huge majorities in states where her opponent had not campaigned because he knew he could not win, trump campaigned hardest in states hardest hit by globalism, and won the electoral college.
If it had been a poker game, Hillary would have been playing against computers, Trump would have been shuffling cards in a dimply lit, smoke filled back room.
Ultimately, an unwavering faith in Big Data has become a sort of religious fundamentalism, an attempt, using only belief and awe of the superior being, to impose order on chaos, an almost fundamentalist approach that borders on statistical fanaticism. Unfortunately, just as mathematics is not a science, neither is statistics, which is more akin to the dark arts that the documented and demonstrable proofs of the natural sciences. It’s an attempt to overlook how little we know and how much we imagine we can control.
With a large segment of the population and an even larger segment of academics, media pundits and politicians eager to reduce human interaction to data points, which enables them to imagine they could control human nature, we run the risk of becoming an increasingly technocratic society where people value safety over stimulation, ease over innovation and convenience over adventure. Ironically, if some Christian preacher talks about The Creation, Nate Silvers fans will scream in unison about the undeniable fact of evolution, thereby only affirming that they don't have a clue how survival of the fittest works. (Hint, it doesn't mean those who work out at the most expensive gym and health club will lead the way to the future.)
RELATED POSTS:
Greenteeth Elsewhere: [
The Original Boggart Blog] ... [
Writerbeat ] ... [
Daily Stirrer.shtml ]...[
Little Nicky Machiavelli]... [
Ian's Authorsden Pages ]... [
It's Bollocks My Dears, All Bollocks ] [
Scribd]...[
Wikinut] ... [
Boggart Abroad] ... [
Grenteeth Bites ] ... [
Latest Posts ] [
Ian Thorpe at Flickr ] ... [
Latest Posts] ... [
Tumblr ] ... [
Authorsden blog ] ... [
Daily Stirrer Headlines]
[
Ian at Facebook ]
