Information aggregation in big data: Wisdom of crowds or stupidity of herds

Abstract

We are entering an age of big data in which our everyday lives depend on tremendous amounts of data and at the same time generate new data. However, the effect of this information convenience on the quality of our decision-making is still not clear. On the one hand, more information is expected to help people make better decisions by serving as the “wisdom of crowds”. On the other hand, imitation among interconnected agents may lead to the “stupidity of herds” with the result that most people will make worse choices. Using agent-based modeling, we explore the information aggregation behaviors of an interconnected population and study how the connectedness among agents influences the checks and balances between the “wisdom of crowds” and the “stupidity of herds”, as well as the decision quality of the agents. We find that in a population of interconnected agents with limited fact-checking capacity, a quasi-equilibrium with a small portion of agents making decisions based on fact checking and a large portion of agents following the majority can be achieved in the process of reinforcement learning. The effects of agents’ fact-checking capacity and search scope on herding behavior, decision quality, and the possibility of systemic failure are also investigated. It is interesting to find that the decision accuracy first increases and then decreases as the agents’ search scope goes up if the agents have a limited fact-checking capacity. This finding implies that a partially connected rather than a fully connected network is preferred from the viewpoint of information aggregation efficiency.