Observability of Errors & the Observer
Some Insights from Dave Dunning and Phil Tetlock on Calibration and Forecasting
Dunning Kruger
The Dunning Kruger effect is shown in the image below and is surprisingly common and resilient:
Those with less "skill" also lack the meta-cognitive skills to be aware of their lack of skill. Quantitatively, this shows up as less skilled people being overconfident about their own performance. This was famously presented in the 1999 paper Unskilled and Unaware of It.
There are a whole host of interesting implications of this -- but for us the key is the advice Dunning offers on how to manage the fact that all of us at some time are in a low skill group and trying to get to a high skill group. We have to develop both expertise and also the meta-cognitive faculties to see how we are actually performing and see our errors.
Dunning's advice ends up being simple, but has the benefit of being anchored in rigorous structured and replicated research — the reason academic frameworks are useful. Meta-cognitive tools to correct for the Dunning-Kruger effect can be as simple as getting feedback from friends or associates with expertise in an area, or being careful of assuming you understand a situation, which is new or unexpected or rare.
In the attached reel I've extracted snippets of Dr. Dunning explaining Dunning Kruger and the problems of expertise and meta-cognition... using the old Greek adage of knowing oneself as the frame for the talks:
Dunning - Cognition
Forecasting & Analysis
Dunning’s admonition to know yourself may sound like a standard recommendation of a psychologist — but we aren’t looking for therapeutic value in his advice. We want to improve our analytical accuracy in making forecasts. You can’t begin to analyze any company or economy until you begin to escape the trap of your own biases, weaknesses and knowledge gaps. And everyone has weaknesses and gaps.
As Dunning states, the individuals with the highest level of expertise get their own skill level right, but they systematically get other people wrong because they can’t imagine not having their level of expertise.
As investors and business analysts our job is to produce superior forecasts. In Phil Tetlock’s terms, ideally we would learn to be Superforecasters. This is a complex idea, but becoming a superforecaster boils down to stripping out your own biases, chunking problems into pieces and thinking in probabilities. Learning to manage the errors you introduce to you analysis yourself is the first step on the road to becoming a forecaster or analyst, on the way to potentially becoming a Superforecaster, whereby you know yourself, and employ the meta-cognitive tools advocated by Dunning to correct the errors of your own design.
The reel below is from Tetlock’s Superforecasting talk at the Long Now Foundation, a few years ago. It briefly introduces some of the broader scope of tools and perspectives that help build groups who are able to collaborate in producing forecasts, which are statistically differentiated from the flip of a coin, or the mythical chimp throwing darts at a list of stocks.
Tetlock-Superforecasting
Closing
The first step to being able to see what’s next in the world, or in any series of events, is to be able to see clearly what has already happened. This is observation or observability — and arguably the hardest part of becoming an observer is learning the tools to know yourself well enough to take yourself out of the equation.
While we have cited Dunning and Tetlock on this, we could also have built the case using excerpts from Peter Drucker’s autobiography, The Adventures of a Bystander. The father of the modern business school identifies himself as a bystander and defines a bystanders as “standing in the wings - much like a fireman in the theater - the bystander sees things neither the actors or audience see. Bystanders reflect — and reflection is a prism rather than a mirror; it refracts.”
You must have a patience and humility to learn to see things neither the actors or audience see, to stand by and seeing what may come — have the confidence to act.
Whether you know yourself better or not, research which fails to reach a conclusion and differentiate from consensus is a waste of time.