Francis Galton & the Wisdom of the Crowds

Francis Galton is a relatively unknown British scientist. It is despite the fact that his contributions to certain concepts that we take for granted is seminal. Galton was a man obsessed with two things: (a) the measurement of physical and mental qualities and (b) breeding. He believed that whenever possible one must quantify and measure. Breeding mattered to Galton because he believed that only a very few people had the characteristics necessary to keep societies healthy. He came to this conclusion after devoting much of his career to measuring these characteristics. As a result, he had little faith in the intelligence of the average person. Galton believed that ‘only if power and control stayed in the hands of the select well-bred few, could a society reman healthy and strong’.

In the year 1906, Galton visited the Annual West of England Fat, Stock and Poultry Exhibition. it was a place where local farmers gathered to apprise the quality of each others cattle and poultry. While he was there, Galton came across a weight judging competition wherein a fat Ox had been selected and placed on display. A large crowd had gathered to place wagers on the weight of the Ox. Actually, they were placing bets on what the weight of the Ox would be after it had been slaughtered and dressed. Anyone who was desirous of placing a bet had to buy a stamped and numbered ticket, fill in his or her name and address, and last of all state their estimate of the weight of the Ox. The guess that was closest to the actual weight when measured would receive the prize. A total of 800 tickets were sold. In other words, 800 people bet on the weight of the Ox. After the contest was over, Galton borrowed the tickets from the organisers and ran the numbers. It so happened that a total of 787 persons actually placed bets. These were arranged in descending order to see if they would form a bell curve. Then he calculated the statistical mean of the guesses which was the collective wisdom of all the farmers about the weight of the Ox. The average so calculated came to 1197 pounds and the Ox actually weighed in at 1198 pounds.

What Francis Galton stumbled upon was that if you put together a big enough and diverse group of people, and ask them to make decisions affecting matters of general interest, the group’s decisions will over time be intellectually superior to any one individual member of the group. The surprising part is that this seems to be the true not sometimes, but always and irrespective of how smart or well informed any one individual member of the group is. The Wisdom of the Crowds can be summarised as follows:

Under the right circumstances, groups are intelligent.
The group estimate is better than the best or the smartest guess.
The group does not have to be dominated by smart people, the collective decision turns out to be the best guess.

The point is that if we can devise a system of crowd sourcing that embodies the three properties listed above, we can draw on the collective intelligence of the crowd, not otherwise. Once the above three conditions have been met, we can glean the useful bits of information that might otherwise stay hidden. As a rule, drawing on collective intelligence makes sense when the collective pool of knowledge is greater than what you would otherwise get access to. In this way we can arrive at a more intelligent and accurate guess than anyone one member of the group would arrive at on his (or her) own.

In today’s internet age, the wisdom of the crowds is at work all around us. Despite the effects of the principle of the Wisdom of the Crowds being so prevalent, it almost seems to be ‘hiding in plain sight’, since most of us are oblivious to it. And when our attention is drawn to the Wisdom of the Crowds, we find it hard to accept. Most of us seem to seek out experts and deliberately ignore the collective wisdom. Francis Galton’s contribution to the statistical world was more than just ‘stumbling upon’ the Wisdom of the Crowds’, as can be seen from the following videos:

One more important caveat is that the Wisdom of the Crowds only works within the context of decision-making. James Surowiecki in his book, ‘The Wisdom Of Crowds’, talks of three kinds of problems and how we address decision-making in each of them. According to him there (a) Cognition Problems (those which have a definitive answer), (b) Coordination Problems (those wherein members try and figure out how to coordinate their behaviour with each other) and (c) Cooperation Problems (those which involve trying to get self-interested but distrustful people to work together). As per Surowiecki, for the ‘Wisdom of the Crowds’ to trounce individual wisdom, the following four conditions must be satisfied:

Diversity of opinion. Each member of the crowd must have some private information.
Independence of the individual opinion which means that the opinion of an individual member of the group should not be influenced by the other members of the group.
Decentralisation, which means that the different members of the group draw on their local knowledge. The members of the group should have different attitudes towards risk, different time horizons, different investing styles and different information. In other words, Cognitive Diversity matters.
Aggregation, which means some mechanism must exist for converting the individual opinions into a collective judgment.

Any group that satisfies the four conditions listed above, is likely to deliver superior and accurate judgments as compared to any individual member of the group who maybe an expert; and this will be true, not sometimes, but always. In other words, expertise and accuracy are unrelated. Why then are so called ‘experts’ given the importance that they are? J. Scott Armstrong, who is a professor at Wharton calls it the ‘Seer-Sucker Theory’ according to which he says that: ‘No matter how much evidence exists that seers do not exist, suckers will always pay for the existence of seers.’ Even if one were to assume that some ‘experts’ are superior to the crowd, the challenge of identifying who that expert is remains. While trying to identify these so called experts, we are more often than not likely to be ‘fooled by randomness’.

The next question is: why does the collective judgment tend to work? It seems that in any such collective decision making group, the individual errors tend to cancel themselves out. If one were to assume that each persons guess has two components; information and error and one were to subtract the error (since it tends to cancel out), what we are left with is information.

Charles Mackay & Madness of Crowds

Charles Mackay published ‘Extraordinary Popular Delusions and the Madness of the Crowds’ in 1852; his original thoughts were published as just Extraordinary Popular Delusions in 1841. Mackay has famously observed: ‘Men, it has been said, think in herds, it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one.’

Mackay would probably have classified the Efficient Market Hypothesis as a delusion if he were alive today! Mackay’s thesis seems to fly in the face of the Wisdom of Crowds thesis, but a closer reading of both of these seemingly contradictory opinions will reveal that they actually complement each other. In other words, we need to understand each one of them separately and see how one complements the other. The video below shows how.

What Mackay understood way back in 1841 is that we need to study human error and then try and understand why it occurs. Mackay was clear that a study of human error may not necessarily inoculate us from the errors that we are prone to commit; but that it does lead to better decision-making or rather it leads to a better understanding of the decision-making process. He has provided us with numerous examples which can help us identify when the wisdom of the crowds is being transformed into the madness of the crowd.

The point to remember is that the madness of a crowd ALWAYS begins with a BIG TRUTH. In other words, what begins as a ‘big truth’ ultimately leads to universal folly and at times (not always) results in crowd madness. Since every crowd always begins with a ‘big truth’, it makes sense to be part of the crowd at an early stage and to stay away from the crowd when the madness starts to build. In the financial markets we have seen time and again that when ‘beliefs’ start having a disproportionate effect, mental short cuts replace rational behaviour.

Crowd psychology has its genesis in our evolution. We have as a species, always lived with high levels of uncertainty. Hence, we seek certainty whenever and wherever we can. As a result, we try and forecast outcomes and then proceed to make decisions, inherently assuming that our forecasts are accurate. In the process we rely on societal clues that often lead us astray. The most intelligent and rational among us search for answers to uncertainty and in the process we disregard our sense of reason. Since prediction and pattern recognition are biological imperatives, most of human folly can be traced to our desperation to find ’cause and effect’ relationships. In the process, we ‘derive’ cause and effect even in places where no such relationships exist. The fact that we are overconfident and very quick to form opinions (and slow to change them), ensures that we respond emotionally and not rationally.

Mackay found that in multiple fields of human endeavour, irrationality actually increases when we are part of a crowd. The reason for this is that, once we are part of the crowd we tend not to think at all. When Mackay wrote his book, the mode of communication was through conventional societal norms. With the advent of ‘digital tribes’ as part of the internet, this seems to be happening all the time. In other words, the recipe for ‘madness’ is actually abetted by the internet. As a species, we are more susceptible now, than we have been at any time in our past. The stage seems to be set for more madness, not less. In other words, we need to ‘look out for the crowd’!

Game Theory & Nash Equilibrium

Game Theory is the study of strategy and decision-making under adverse conditions. When the concept was first introduced, its application was restricted to zero-sum games. Today the application of the principles of Game Theory are widespread and the words ‘game theory’ is used as an ‘umbrella term’ for the science of logical decision-making.

For our purposes a ‘game‘ will mean a situation that has three components:

a set of players characterised by people
a set of ‘moves’ (popularly called strategies) that each of the players can make and
a ballpark calculation that each of the players has about the outcomes of their moves which are mathematically described as payoffs

Game Theory is about more than just board games and gambling. It is a branch of economics that considers how players of a mathematically formalised game can optimise their decisions. The crucial part in all of these so-called ‘games’ is that, what one person does has an effect on the other players. As such, Game Theory teaches us decision-making when conditions are interdependent on one another. The main purpose behind learning Game Theory is to meet two objectives, these are:

to help students recognise strategic situations and
to provide a framework whereby one can make better decisions by changing the game in such a way that a no-win situation can result in mutually beneficial outcomes.

Before we dive into Game Theory, we need to understand the concept of ‘strategy’. While discussing Game Theory, the term ‘Strategy’ is defined as ‘any one of the options which any one of the players chooses in a setting where the outcome of his or her decision will be decided not by his or her decision alone, but by the decisions of the other players.’

The most well known ‘game’ in Game Theory is the ‘Prisoners Dilemma.’ Albert Tucker formalised this game and gave it its name. It can be used to illustrate the important Game Theory concepts. The set up for the Prisoners Dilemma game is as follows:

Two criminals, A and B are arrested and put in jail, each in their own cell with no way to communicate. The assumption is that for each player the choices are restricted to those provided in the game and there is no provision for any negotiation of any kind,
The Prosecutor doesn’t have enough evidence to convict either one for a major crime, but does have enough to convict both for minor infractions.
However, if the prosecutor could get one of the prisoners to turn on their co-conspirator, the other one could be put away for the major crime. Hence, they offer each prisoner a bargain. Each prisoner can either (a) betray the other by testifying that the other committed the crime, or (b) to cooperate with the other by remaining silent. The offer is:
- If A and B each betray the other, each of them serves 2 years in prison
- If A betrays B, but B remains silent, A will be set free and B will serve 3 years in prison and vice-versa
- If A and B both remain silent, both of them will only serve one year in prison.

Given the above payoff matrix, for a player who is acting rationally and independently, the dominant strategy is to always betray the other prisoner. That’s the only way of getting off free. The rub is that, the other prisoner in all probability, also ends up following the same strategy. As a result, the outcome is a mutual betrayal and both the prisoners go to prison for a longer period than if both had remained silent. Hence the ‘dilemma’ in the game: should one risk betrayal or should one repose trust in the other prisoners solidarity? (which can be reframed as should one compete or should one collude?).

The dual betrayal is known as the Nash equilibrium of this game. It is named after John Nash, a mathematician and one of the pioneers of Game Theory and the subject of the biopic A Beautiful Mind. In the game above, the Nash equilibrium is the strategy of dual betrayals, because if either player instead chose to remain silent, that player would get a longer sentence. To both get a shorter sentence, they’d have to act cooperatively, coordinating their strategies. Such a coordinated strategy is unstable (i,e, it is not an equilibrium), because either player could then betray the other, to better their outcome. In any game that you play you want to know whether there is a Nash equilibrium, as that is the most likely outcome unless something is done to change the rules of the game. In other words, a Nash equilibrium is a situation in which no person can improve his or her payout, given what others are doing. Given what the other players are doing, you are making the best possible choice. A Nash equilibrium is not necessarily socially optimal, nor is it the best for the players; it is the best response under the circumstances. At times the Nash equilibrium will seem fair and at times it may not. Theoretically speaking, there can be several Nash equilibria for each game.

When we undertake a study of Behavioral Finance we are trying to learn to be more rational. Being rational is defined as someone who has reason or understanding. What makes the Prisoners Dilemma so fascinating is that in the above matrix if both the players were to behave rationally (which is to betray each other), it does not lead to the best overall outcome. Had both the players behaved irrationally and stayed silent, they would have served only one year each. In other words, by behaving in a rational manner, the outcome is sub-optimal. In such a scenario, where both the prisoners realise that betrayal is the optimal individual strategy and there is no gain to unilaterally changing it, they arrive at the conclusion which is known as the ‘Nash equilibrium’. In the above example, mutual betrayal is the only Nash equilibrium. The essential features of a Nash equilibrium are:

a set of strategies, one for each player, such that no player has an incentive to change his or her strategy given what the other players are doing.
a set of player choices, wherein a change of strategy by any one player would worsen their outcome.

Prisoners Dilemma is a simple yet powerful model of the Game Theory. The takeaways from this game in the ‘real world’ are as follows:

that the rational solution for a company is not always the solution that is optimal for the industry as a whole.
that competitive markets need not be zero-sum.
when the conditions exist, executives can make pricing and capacity decisions that maximise long-term value.
investors can be the judge of how the game is being played and select potential winners.

The question that we need to ask is what happens if the above game is played repeatedly? In such a scenario, the prisoners will choose to remain silent; they will cooperate and it would mean that collusion (and not competition) will be the outcome of the game. The practical application of the above is that in a competitive marketplace, firms might choose to collude rather than to compete; since if firms collude, the outcome is higher profits. This interpretation flies in the face of traditional economic theory, since economists argue that firms would always choose to compete rather than to collude.

In the above example, we could introduce a real world twist and that is to factor in a choice that the players are given. Hence, assuming that the players are given a choice; they may choose to cooperate with each other or they may choose not to cooperate with each other. In such a scenario, mutual cooperation yields a better outcome than mutual defection. As a result, today the concept is split between two factions; the cooperative game theory and the non-cooperative game theory. Non-cooperative game theory tries to predict strategies and payoffs in order to find their Nash equilibria. As opposed to this, cooperative game theory focusses on predicting which group of players will collude and improve their collective payoffs. In the real world, we see both the variations of game theory being applied depending upon the bargaining powers and the needs of the different players.

To conclude, Game Theory is the study of interactions among players trying to maximise their payoff. What makes the analysis tricky is that the actions (and reactions) of the players determine the payoff. As a result, game theory forces executives to think not only about their own choices, but also how those choices will affect the choices of their competitors. Not all executives naturally put themselves in their competitors shoes. The conceptual framework behind Game Theory can be easily adapted to many real-life situations. Today, Game Theory is widely applied in various fields like political science, business, evolutionary biology, philosophy and computer science.