Superforecasting by Tetlock

Ref: Phillip Tetlock (2015). Superforecasting: The Art and Science of Prediction. Crown Publishing.

___________________________________________________________________________

Summary­

• A researcher gathered a big group of experts—academics, pundits, and the like—to make thousands of predictions about the economy, stocks, elections, wars, and other issues of the day. Time passed, and when the researcher checked the accuracy of the predictions, he found that the average expert did about as well as random guessing. In this experiment, it was easiest to beat chance on the shortest-range questions that only required looking one year out, and accuracy fell off the further out experts tried to forecast—approaching the dart-throwing-chimpanzee level three to five years out.

• Baseball managers wouldn’t dream of getting out the checkbook to hire a player without consulting performance statistics. Even fans expect to see player stats on scoreboards and TV screens. And yet when it comes to the forecasters who help us make decisions that matter far more than any baseball game, we’re content to be ignorant.

• Every day, the news media deliver forecasts without reporting, or even asking, how good the forecasters who made the forecasts really are.

• Forecasting: Estimating the likelihood of something happening. Forecasting is not a “you have it or you don’t” talent. It is a skill that can be cultivated. This book will show you how. I believe it is possible to see (forecast) into the future, at least in some situations and to some extent, and that any intelligent, open-minded, and hardworking person can cultivate the requisite skills.

• Superforecasting: Predictive forecasting based on open-minded, careful, curious, numerate, and, above all, self-critical analysis.

___________________________________________________________________________

Illusion of Knowledge

• Our judgments about risks—Does gun control make us safer or put us in danger?—are driven less by a careful weighing of evidence than by our identities, which is why people’s views on gun control often correlate with their views on climate change, even though the two issues have no logical connection to each other. Psycho-logic trumps logic. And when Kahan asks people who feel strongly that gun control increases risk, or diminishes it, to imagine conclusive evidence that shows they are wrong, and then asks if they would change their position if that evidence were handed to them, they typically say no. That belief block is holding up a lot of others. Take it out and you risk chaos, so many people refuse to even imagine it.

• Psychologists who test police officers’ ability to spot lies in a controlled setting find a big gap between their confidence and their skill. And that gap grows as officers become more experienced and they assume, not unreasonably, that their experience has made them better lie detectors. As a result, officers grow confident faster than they grow accurate, meaning they grow increasingly overconfident.

• It’s easy to impress people by stroking your chin and declaring “There is a 73% probability Apple’s stock will finish the year 24% above where it started.” Toss in a few technical terms most people don’t understand—“stochastic” this, “regression” that—and you can use people’s justified respect for math and science to get them nodding along. This is granularity as bafflegab. It is unfortunately common.

• It’s easy to misinterpret randomness. We don’t have an intuitive feel for it. Randomness is invisible from the tip-of-your-nose perspective.

• Anyone can easily “predict” the next stock market crash by incessantly warning that the stock market is about to crash. And yet many people take these hollow claims seriously.

• The accuracy of expert predictions declines toward chance 5y out. And yet, this sort of forecasting is common, even within institutions that should know better.

• The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction, and that is enough.

• God Complex: An illusion of knowledge; the idea that your judgement alone reveals the truth (Cochrane).

___________________________________________________________________________

Good Judgement Project

Good Judgement Project: www.goodjudgment.com; launched by the author using volunteers to forecast the future; two key conclusions emerged from the research: 1) foresight is real, 2) people can learn to forecast.

• Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs.

• Thanks to IARPA, we now know a few hundred ordinary people and some simple math can not only compete with professionals supported by a multibillion-dollar apparatus but also beat them.

• Across all 4y of the tournament, superforecasters looking out 300d were more accurate than regular forecasters looking out 100d.

• On average, teams were 23% more accurate than individuals. Teams of ordinary forecasters beat the wisdom of the crowd by about 10%. Prediction markets beat ordinary teams by about 20%. And Superteams beat prediction markets by 15% to 30%.

• Brier Scores: Measure the distance between what you forecast and what actually happened. 0 is a perfect score.

___________________________________________________________________________

Forecaster Types

• Hedgehogs: Confident forecasters with a single perspective analysis; pile up reasons why they are right- “furthermore,” “moreover”- without considering other perspectives and the doubts and caveats they raise. Hedgehogs tell tight, simple, clear stories that grab and hold audiences. They are likelier to say something definitely will or won’t happen.

  • The hedgehog’s one Big Idea doesn’t improve his foresight. It distorts it. And more information doesn’t help because it’s all seen through the same tinted glasses. It may increase the hedgehog’s confidence, but not his accuracy. That’s a bad combination. The predictable result? When hedgehogs made forecasts on the subjects they knew the most about—their own specialties—their accuracy declined.

• Foxes: Careful, less confident forecasters, that are less likely to say something is “certain” or “impossible,” and are likelier to settle on shades of “maybe.” Their stories are complex, full of “howevers” and “on the other hands,” because they look at problems one way, then another, and another. This aggregation of many perspectives is good, essential, forecasting. Foxes seek out information and perspectives from multiple sources that they synthesize into a single conclusion.

• So why does one group do better than the other in forecasting? It isn’t whether they had PhDs or access to classified information. Nor was it what they thought—whether they were liberals or conservatives, optimists or pessimists. The critical factor is how they thought.

• A key reason why the foresight of foxes is superior to that of hedgehogs with their green-tinted glasses. Foxes aggregate perspectives.

___________________________________________________________________________

Systems 1 & 2

• System 1 (S1): What You See Is All There Is (WYSIATI) (‘Tip of your Nose View’); automatic perceptual and cognitive operations undertaken quickly by the brain; S1 follows a primitive psycho-logic: if it feels true, it is. S1 is designed to jump to conclusions from little evidence.

  • Tip-of-Your-Nose Perspective: S1; insists that what we see is reality objectively and correctly, so there is no need to consult other perspectives.

• System 2 (S2): Conscious and rational thought undertaken slowly by the brain on what we choose to focus on.

• S1 comes first. It is fast and constantly running in the background. If a question is asked and you instantly know the answer, it sprang from S1. S2 is charged with interrogating that answer. Does it stand up to scrutiny? Is it backed by evidence? This process takes time and effort, which is why the standard routine in decision making is this: first S1 delivers an answer, and only then can S2 get involved, starting with an examination of what S1 decided.

• Confirmation Bias: Our natural inclination is to grab on to the first plausible explanation and happily gather supportive evidence without checking its reliability.

___________________________________________________________________________

Probabilistic Thinking

• Most people only have three settings: “gonna happen,” “not gonna happen,” and “Maybe.”

  • A three-setting dial gives quick, clear directions. Is that a lion? YES = run! MAYBE = stay alert! NO = relax.

• Evolutionary speaking, the ability to distinguish between a 60% probability and an 80% probability adds little. In fact, a more fine-grained analysis could slow you down—and get you killed.

• Study after study shows that people attach very different meanings to probabilistic language like “could,” “might,” and “likely. Something that is “possible” has a likelihood ranging from almost zero to almost 100%.

• Both 0% and 100% weigh far more heavily in our minds than economists say they should.

• Most people translate a high probability into ‘this will happen’.

• Every “may” is accompanied by an asterisk and the words “or may not” are buried in the fine print. “Maybe” has to be subdivided into degrees of probability.

• By rejecting certainty, everything becomes a matter of probability

• “The probability that I would meet the love of my life was tiny. But it happened. So it was meant to be. Therefore, the probability that it would happen was 100%.” This is beyond dubious. It’s incoherent. Logic and psycho-logic are in tension.

• A probabilistic thinker will be less distracted by “why” questions and focus on “how.” This is no semantic quibble. “Why?” directs us to metaphysics; “How?” sticks with physics.

• Probabilistic thinkers take the outside view toward even profoundly identity-defining events, seeing them as quasi-random draws from distributions of once-possible worlds.

• The more a forecaster inclined toward it-was-meant-to-happen thinking, the less accurate her forecasts were. Or, put more positively, the more a forecaster embraced probabilistic thinking, the more accurate she was.

• Finding meaning in events is positively correlated with wellbeing but negatively correlated with foresight.

___________________________________________________________________________

Science

• Science doesn’t tackle “why” questions about the purpose of life. It sticks to “how” questions that focus on causation and probabilities.

• In practice, scientists do use the language of certainty, but only because it is cumbersome whenever you assert a fact to say “although we have a substantial body of evidence to support this conclusion, and we hold it with a high degree of confidence, it remains possible, albeit extremely improbable, that new evidence or arguments may compel us to revise our view of this matter.”

• The best evidence that a hypothesis is true is often an experiment designed to prove a hypothesis is false, but which fails to do so.

• Scientists must be able to answer the question “What would convince me I am wrong?” If they can’t, it’s a sign they have grown too attached to their beliefs. The key is doubt. Scientists can feel just as strongly as anyone else that they know The Truth. But they know they must set that feeling aside and replace it with finely measured degrees of doubt—doubt that can be reduced (although never to zero) by better evidence from better studies.

___________________________________________________________________________

Mindset

• In one experiment, Dweck scanned the brains of volunteers as they answered hard questions, then were told whether their answers were right or wrong and given information that could help them improve. The scans revealed that volunteers with a fixed mindset were fully engaged when they were told whether their answers were right or wrong but that’s all they apparently cared about. Information that could help improve their answers didn’t engage them. “Even when they’d gotten an answer wrong, they were not interested in learning what the right answer was,” Dweck wrote. “Only people with a growth mindset paid close attention to information that could stretch their knowledge. Only for them was learning a priority.”

• Growth Mindset: Believing that your abilities are largely the product of effort—that you can “grow” to the extent that you are willing to work hard and learn (Dweck).

• Fixed Mindset: The belief that we are who we are, and abilities can only be revealed, not created and developed.

  • People with the fixed mindset say things like “I’m bad at math” and see that as an immutable feature of who they are, like being left-handed or female or tall. This has serious consequences. The person who believes he is bad at math, and always will be, won’t try hard to improve, because that would be pointless, and if he is compelled to study math—as we all are in school—he will take any setback as further proof that his limits have been revealed and he should stop wasting his time as soon as possible. Whatever potential he had for improvement will never be realized. Thus, the belief “I am bad at math” becomes self-fulfilling.

___________________________________________________________________________

Feedback

• Effective learning can’t happen without clear feedback, and you can’t have clear feedback unless your forecasts are unambiguous and scorable.

• Not all practice improves skill. It needs to be informed practice. You need to know which mistakes to look out for—and which best practices really are best. Effective practice and experience is not enough. It must be accompanied by clear and timely feedback.

• Calibration: How closely your confidence matches your accuracy; research routinely finds that people are overconfident.

• Perpetual Beta: A computer programming term for a program that is not intended to be released in a final version but will, instead be used, analyzed, and improved without end.

• Try, fail, analyze, adjust, try again: cycle through these steps ceaselessly.

___________________________________________________________________________

Groups

• Groupthink: The tendency of members of any small cohesive group to maintain esprit de corps by unconsciously developing a number of shared illusions and related norms that interfere with critical thinking and reality testing (Janis). Groups that get along too well don’t question assumptions or confront uncomfortable facts, so everyone agrees, which is pleasant; the fact that everyone agrees is tacitly taken to be proof the group is on the right track.

• The dual-edged nature of working in groups: teams can cause terrible mistakes or they can sharpen judgment and accomplish together what cannot be done alone.

• Cognitive Loafing: The tendency of groups to assume others will conduct the complex problem solving.

• Aggregation of data and answers within a group is critical to accuracy; aggregation can only do its magic when people form judgments independently.

• Be cooperative but not deferential. Consensus is not always good; disagreement not always bad. If you do happen to agree, don’t take that agreement—in itself—as proof that you are right. Never stop doubting. Pointed questions are as essential to a team as vitamins are to a human body. On the other hand, the opposite of groupthink—rancor and dysfunction—is also a danger. Team members must disagree without being disagreeable. Practice “constructive confrontation,” to use the phrase of Andy Grove, the former CEO of Intel. Precision questioning is one way to do that.

• Tactfully dissect the vague claims people often make.

• In groups, bring in outsiders, suspend hierarchy, and keep the leader’s views under wraps.

• How did superteams do so well? By avoiding the extremes of groupthink and Internet flame wars. And by fostering mini-cultures that encouraged people to challenge each other respectfully, admit ignorance, and request help. Superteams used language like “our” more than “me” or “I”.

• Group Personas (Adam Grant- Wharton)

  • Givers: Those who contribute more to others than they receive in return.

  • Matchers: Those who give as much as they get.

  • Takers: Those who contribute less than they take.

___________________________________________________________________________

Leadership

• Leaders must be reasonably confident, and instill confidence in those they lead, because nothing can be accomplished without the belief that it can be. Decisiveness is another essential attribute. Leaders can’t ruminate endlessly. They need to size up the situation, make a decision, and move on. And leaders must deliver a vision—the goal that everyone strives together to achieve.

• How can leaders be confident, and inspire confidence, if they see nothing as certain? How can they be decisive and avoid “analysis paralysis” if their thinking is so slow, complex, and self-critical? How can they act with relentless determination if they readily adjust their thinking in light of new information or even conclude they were wrong?

___________________________________________________________________________

Superforecasting

Superforecasting: Predictive forecasting based on open-minded, careful, curious, numerate, and, above all, self-critical analysis.

• Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others—and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.

• Timeline: Forecasts must have clearly defined terms and timelines. They must use numbers. The further out the forecaster tries to look, the more opportunity there is for chaos to flap its butterfly wings and blow away expectations.

• Anchoring: The first number given in an estimate. Others “anchor” off of it when making predictions. A forecaster who starts by diving into the inside view risks being swayed by a number that may have little or no meaning. But if she starts with the outside view, her analysis will begin with an anchor that is meaningful. And a better anchor is a distinct advantage.

• Feedback: Superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence.

  • Superforecasters update more often than other forecasters, they update in smaller increments.

• Reaction: There are two dangers a forecaster faces after making the initial call. One is not giving enough weight to new information- underreaction. The other danger is overreacting to new information, seeing it as more meaningful than it is, and adjusting a forecast too radically.

• Precision: Be precise; granularity predicts accuracy: the average forecaster who sticks with the tens—20%, 30%, 40%—is less accurate than the finer-grained forecaster who uses fives—20%, 25%, 30%—and still less accurate than the even finer-grained forecaster who uses ones—20%, 21%, 22%.

• Synthesis: Use the dragonfly eye to aggregate irreducibly subjective judgements into a single image (thesis + antithesis = synthesis). Look at problems from the perspectives of both logic and psycho-logic, and combine what you find.

  • Dragonfly Eye: Pursuit of point-counterpoint discussions routinely.

  • Wisdom of the Crowd: Combining the judgements of a large group of people.

Method

1. Triage

2. Break seemingly intractable problems into tractable sub-problems (Fermi-ize).

3. Strike the right balance between inside and outside views.

4. Strike the right balance between under- and overreacting to evidence.

5. Look for the clashing causal forces at work in each problem.

6. Strive to distinguish as many degrees of doubt as the problem permits but no more.

7. Strike the right balance between under- and overconfidence, between prudence and decisiveness.

8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.

9. Bring out the best in others and let others bring out the best in you.

10. Master the error-balancing bicycle.

11. Don’t treat commandments as commandments.

Traits of the Superforecaster

• Cautious: Nothing is certain.

• Humble: Reality is infinitely complex.

• Nondeterministic: What happens is not meant to be and does not have to happen. 

• Actively Open-Minded: Beliefs are hypotheses to be tested, not treasures to be protected.

• Intelligence and Knowledgeable, with a Need for Cognition: Intellectually curious, enjoy puzzles and mental challenges.

• Reflective: Introspective and self-critical.

• Numerate: Comfortable with numbers in their methods of forecasting.

• Pragmatic: Not wedded to any idea or agenda.

• Analytical: Capable of stepping back from the tip-of-your-nose perspective and considering other views.

• Dragon-Eyed: Value diverse views and synthesize them into their own.

• Probabilistic: Judge using many grades of maybe.

• Thoughtful Updaters: When facts change, they change their minds.

• Good intuitive Psychologists: Aware of the value of checking thinking for cognitive and emotional biases In their work ethic.

• Growth Mindset: Believe it’s possible to get better.

• Grit: Determined to keep at it however long it takes.

___________________________________________________________________________

Predictive Algorithms

• 1954: Psychologist, Paul Meehl, wrote a small book that reviewed 20 studies showing that well-informed experts predicting outcomes—whether a student would succeed in college or a parolee would be sent back to prison—were not as accurate as simple algorithms that added up objective indicators like ability test scores and records of past conduct.

• Research (>200 studies today) has shown that in most cases statistical algorithms beat subjective judgment.

• When you have a well-validated statistical algorithm, use it.

___________________________________________________________________________

Resources

• Gates Foundation: One of the world’s largest foundations; renowned for rigorous evaluation of its programs. https://www.gatesfoundation.org/

• Good Judgement Project: www.goodjudgment.com

• Intelligence Advanced Research Projects Activity (IARPA): https://www.iarpa.gov/; An agency within the IC that reports to the director of National Intelligence; its job is to support daring research that promises to make American intelligence better.

• Office of the Director of National Intelligence (ODNI): https://www.dni.gov/; leads the network of 16 intelligence agencies.  

• Quadrennial Defense Review: https://history.defense.gov/Historical-Sources/Quadrennial-Defense-Review/; a 20y forecast of the national security environment; required by congress every four years.

___________________________________________________________________________

Misc Quotes

“Only when we are proven wrong so clearly that we can no longer deny it to ourselves will we adjust our mental models of the world—producing a clearer picture of reality.”

“We can’t go on living this way.”-Gorbachev to his wife Raisa (the day before his ascension to power).

“Too vague to judge; that’s typical of the answers to big, important questions like ‘How does this all turn out?’”

“Contemplating the 100B known stars in our galaxy and the 100B known galaxies instills profound humility.”

“If you aimlessly examine one tree, then another, and another, you will quickly become lost in the forest.”

“I want to hear from a one-armed economist because I’m sick of hearing “on the one hand…on the other.”- Truman.

“Historical probabilities—all the possible ways the future could unfold—are distributed like wealth, not height. That means our world is vastly more volatile than most of us realize and we are at risk of grave miscalculations.”-Taleb.

“Our beliefs about ourselves and the world are built on each other in a Jenga-like fashion.”

“An array of judgments is welcome proof that the people around the table are actually thinking for themselves and offering their unique perspectives.”

“We frequently pass through phases of history riddled with irreducible uncertainty—phases in which luck trumps skill. These are the moments when the laws of statistics tell us to shift bets and put our money on massive regression to the mean.”

“Never tell people how to do things, tell them what to do, and they will surprise you with their ingenuity.”-General George Patton.

“It is wise to take admissions of uncertainty seriously, but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”-Daniel Kahneman.

“Plans are merely a platform for change.”-Israeli Defense Forces slogan.

“Japanese Americans may well be the Achilles’ heel of the entire civilian defense effort.”-CA AG Earl Warren.

“Politics, defined broadly, is nothing more than a struggle for power; ‘kto, kogo’; “who, whom”, shorthand for “Who does what to whom?”-Vladimir Lenin.

“Quit pretending you know things you don’t and start running experiments. Give the training to one randomly chosen group of forecasters but not another. Keep all else constant. Compare results.”-Archie Cochrane.

“The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.”

“Any sufficiently advanced technology is indistinguishable from magic.”-Arthur C. Clarke. 

“In this situation, what you started to get was probabilities that disguised uncertainty as opposed to actually providing you with useful information.”-Obama (on killing Bin Laden).

“After 9/11, the IC was blamed for failing to connect the dots and underestimating the risk of terrorist attacks. After the 2003 invasion of Iraq, the IC was blamed for connecting unconnected dots and overestimating the risk of WMD.”

“All who drink of this treatment recover in a short time, except those whom it does not help, who all die. It is obvious, therefore, that it fails only in incurable cases.”-Galen.

“It is I, and I alone, who has revealed the true path of medicine.”-Galen.

“I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal….This may seem basic, but it is amazing how often it is not done and how hard it is to get right.”-Bill Gates.

“The severe economic turmoil of 2008 and 2009 generated a fierce policy debate. Two camps coalesced: the Keynesians and the Austerians. The Keynesians called for heroic improvisation by central banks and aggressive deficit spending by governments. The Austerians called for austerity—government budget cuts—and warned that the Keynesian policies would ignite inflation and sink currencies.”

“Legend says that while conferring with Roosevelt at Quebec, Churchill sent Keynes a cable reading, ‘Am coming around to your point of view.’ His Lordship replied, ‘Sorry to hear it. Have started to change my mind.”

“It’s funny that a 90% chance of failing people don’t like, but a 10% chance of changing the world people love.”

“In noisy public arenas, strident voices dominate debates, and they have zero interest in adversarial collaboration.”

“Intellectual humility compels the careful reflection necessary for good judgment; confidence in one’s abilities inspires determined action.”

“The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function.”-F. Scott Fitzgerald.

“Our minds crave certainty and when they don’t find it, they impose it.”

“Let’s not make the cynic’s mistake of thinking that those who dominate debates are the only debaters. We hear them most because they are the loudest and the media reward people who shout into bullhorns.”

“Despite the evil nature of the regime that it served, it must be admitted that the Germany Army of World War II was, man for man, one of the most effective fighting forces ever seen.”-Historian James Corum.

“The archetypal academic; theory-poisoned and indecisive.”

“The wise officer knows the battlefield is shrouded in a fog of uncertainty…at least one thing must be certain: one’s own decision. One must adhere to it and not allow oneself to be dissuaded by the enemy’s actions until this has become unavoidably necessary.”-Moltke.

“Once a course of action has been initiated it must not be abandoned without overriding reason; in the changing situations of combat, inflexibly clinging to a course of action can lead to failure.”-Wehrmacht Manual.

“Today, feelings are masquerading as knowledge.”

“The Prussian military had long appreciated uncertainty—they had invented board games with dice to introduce the element of chance missing from games like chess.”

“Everything is uncertain.”-Moltke.

“(Never trust your plan as) no plan of operations extends with certainty beyond the first encounter with the enemy’s main strength.”-Moltke.

___________________________________________________________________________

Terminology

• Aleatory Uncertainty: Something you don’t know and is unknowable; ensures life will always have surprises, regardless of how carefully we plan.

• Anchoring: The first number given in an estimate. Others “anchor” off of it when making predictions.

• Auftragstaktik: Mission Command (German); The idea that “War cannot be conducted from the green table, frequent and rapid decisions can be shaped only on the spot according to estimates of local conditions (Moltke).” Decision-making power must be pushed down the hierarchy so that those on the ground—the first to encounter surprises on the evolving battlefield—can respond quickly. Auftragstaktik blends strategic coherence and decentralized decision making with a simple principle: commanders are to tell subordinates what their goal is but not how to achieve it.

• Base Rate: How common something is within a broader class.

• Bayes Theorem (1761): P(H|D)/P(-H|D) = P(D|H)/P(D|-H) • P(H)/P(-H), Posterior Odds = Likelihood Ratio • Prior Odds. In simple terms, the theorem says that your new belief should depend on two things—your prior belief (and all the knowledge that informed it) multiplied by the “diagnostic value” of the new information.

• Belief Perseverance: The intransigence of people to acknowledge new information that upsets their settled beliefs.

• Black Swan Event: A highly improbable consequential event that we can’t imagine it until we experience it. Black swans, and black swans alone, determine the course of history; “History and societies do not crawl, they make jumps (Taleb).”

• Corpus Callosum: The connection between the right and left hemispheres of the brain.

  • Severing the corpus callosum is a traditional treatment for severe epilepsy.

• Counterfactual Thinking: Thinking about how something might have turned out differently than it actually did.

• Crowd Within: Asking people to assume their initial judgement is wrong, to seriously consider why that might be, and then make another judgement to produce a second estimate which, when combined with the first, improves accuracy almost as much as getting a second estimate from another person. The same effect is produced simply by letting several weeks pass before asking people to make a second estimate.

• Efficient Market Hypothesis (EMH): Markets are perfect, or such efficient aggregators of information that no mortal should ever be so foolish as to aspire to beat them.

• Epistemic Uncertainty: Something you don’t know but is, at least in theory, knowable.

• Fate: The view that events happen for a reason and that there is an underlying order to life that determines how events turn out.

• Genius: Top 1% of IQ Tests; >135.

• Grit: Passionate perseverance of long-term goals, even in the face of frustration and failure. Married with a growth mindset, it is a potent force for personal progress.

• Hindsight Bias: “I knew it all along”; once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome.

  • Putting yourself in the position of someone at the time and not being swayed by your knowledge of what happened later is a struggle.

• Intellectual Humility: A recognition that reality is profoundly complex; that seeing things clearly is a constant struggle.

• Intelligence Analysis: The methodical examination of the information collected by spies and surveillance to figure out what it means, and what will happen next.

• National Intelligence Estimates: The consensus view of the CIA, NSA, DIA, and 13 other agencies.

• Paradox of Success (CEO Disease): Success can lead to acclaim that can undermine the habits of mind that produced the success. Such hubris often afflicts highly accomplished individuals.

• Pattern Recognition: With training or experience, people can encode patterns deep in their memories in vast number and intricate detail.

  • “If I study a position for an hour then I am usually going in loops and I’m probably not going to come up with something useful. I usually know what I am going to do after 10 seconds; the rest is double-checking.”

• Pointillism: Painting by dabbing tiny dots on the canvas.  

• Quants: Math Wizards on Wall Street.

• Regression to the Mean: A phenomenon where if one sample of a random variable is extreme, the next sampling of the same random variable is likely to be closer to its mean. Slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance.

• Scope: The time frame of a forecast.

• Scylla & Charybdis: In Greek mythology, any discussion of two opposing dangers called for Scylla and Charybdis. Scylla was a rock shoal off the coast of Italy. Charybdis was a whirlpool on the coast of Sicily, not far away. Sailors knew they would be doomed if they strayed too far in either direction.

• Snellen Eye-Chart: The standard eye exam chart used to test vision.   

• What You See Is All There Is (WYSIATI): The mother of all cognitive illusions; the egocentric worldview that prevents us from seeing any world beyond the one visible from the tips of our noses.

• Wisdom of the Crowd: Combining the judgements of a large group of people.

___________________________________________________________________________

People

• Aelius Galenus ‘Galen’ (129-216): Greek physician, surgeon, and philosopher with Roman citizenship, considered one of the most accomplished of all medical researchers of antiquity. Galen influenced the development of various scientific disciplines, including anatomy, physiology, pathology, pharmacology, neurology, philosophy, and logic. Galen’s understanding of anatomy and medicine was principally influenced by the then-current theory of the four humors: black bile, yellow bile, blood, and phlegm, as first advanced by the author of “On the Nature of Man” in the Hippocrates corpus (Wiki).

• Eric Arthur Blair ‘George Orwell’ (1903-1950): English novelist, essayist, journalist, and critic. Orwell is characterized by lucid prose, social criticism, opposition to totalitarianism, and support of democratic socialism. Notable works include “Animal Farm” (1945) and “1984” (1949) (Wiki).

  • In his famous essay “Politics and the English Language,” Orwell concluded with six emphatic rules, including “never use a long word where a short one will do” and “never use the passive where you can use the active.” But the sixth rule was the key: “Break any of these rules sooner than saying anything outright barbarous.”

• Archibald ‘Archie’ Cochrane (1909-1988): Scottish doctor noted for his book “Effectiveness and Efficiency: Random Reflections on Health Services” which advocated the use of randomized control trials to make medicine more effective and efficient. His advocacy eventually led to the development of the Cochrane Library database of systematic reviews. He is known as the father of modern clinical epidemiology and evidence- based medicine and is considered to be the originator of the idea of evidence- based medicine in the current era (Wiki).

___________________________________________________________________________

Chronology

• 18 Sep, 2014: The Scottish people vote NO (55.3% to 44.7%) in a referendum on leaving the UK.-Superforecasting by Tetlock.

• 2011: A VBIED kills 8 people and injures >200 more in Oslo.-Superforecasting by Tetlock.

• 2011: Anders Breivik kills dozens at a summer camp for young people run by Norway’s ruling Labour Party. Breivik felt Norway had betrayed him with its multicultural policies.-Superforecasting by Tetlock.

• 17 Dec, 2010- Dec, 2012: Arab Spring; protests erupt in Egypt, Libya, Syria, Jordan, Kuwait, and Bahrain. After three decades in power, the Egyptian dictator Hosni Mubarak is driven from office. Elsewhere, protests swell into rebellions, rebellions into civil wars.-Superforecasting by Tetlock.

  • 14 Jan, 2011: Tunisian President Ben Ali flees to exile in Saudi Arabia, ending his 23y kleptocracy.-Superforecasting by Tetlock.

  • 17 Dec, 2010: Self-Immolation of Tunisian 26yo Mohmed Bouazizi, sparks protests that lead to the Arab Spring. Police respond with undue force. Hoping to assuage the public, the dictator of Tunisia, President Zine el-Abidine Ben Ali, visits Bouazizi in the hospital (Bouazizi dies on 4 Jan, 2011).-Superforecasting by Tetlock.

• 2006: Formation of the Intelligence Advanced Research Projects Activity (IARPA); its mission is to fund cutting-edge research with the potential to make the intelligence community smarter and more effective.-Superforecasting by Tetlock.

• 2006: Murder of former Russian spy Alexander Litvinenko, a critic of Putin, in London with Po-210.-Superforecasting by Tetlock.

• 2002: The US IC issues NIE 2002-16HC assessing Iraq as a country capable of carrying out a nuclear strike within a very short period of time (Wiki). Postmortems on NIE 2002-16HC later reveal that the IC had never seriously explored the idea that it could be wrong. There were no ‘red teams’ to attack the prevailing views, no analyses from devil’s advocates, no papers that provided competing possibilities.-Superforecasting by Tetlock.

• 1998: The US FAA examine a scenario in which terrorists hijack FedEx cargo planes and crash them into the World Trade Center.-Superforecasting by Tetlock.

• 1994: A plot to hijack a jet and crash it into the Eiffel Tower is broken up.-Superforecasting by Tetlock.

• 1991: A Russian Coup fails to overthrow Gorbachev.-Superforecasting by Tetlock.

• 1985: Death of short-serving Soviet leader Konstantin Chernenko. Within hours, the Politburo appoints Mikhail Gorbachev, an energetic and charismatic 54yo as the next general secretary of the USSR CCP. Gorbachev introduces policies of glasnost (openness) and perestroika (restructuring) to liberalize the USSR. Gorbachev also seeks to normalize relations with the US and reverse the arms race.-Superforecasting by Tetlock.

• Jun, 1982: Millions of people across the Western world march against the arms race, including 700K Americans march in NYC.-Superforecasting by Tetlock.

• 1982: Death of long-serving Soviet leader Leonid Brezhnev. He is replaced by a frail old man who soon dies and is replaced by another, Konstantin Chernenko, who is also expected to die soon.-Superforecasting by Tetlock.

• 1972: Rise of Chaos Theory following the publication of “Predictability: Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?” by American meteorologist Edward Lorenz: A decade earlier, Lorenz had discovered by accident that tiny data entry variations in computer simulations of weather patterns- like replacing 0.506127 with 0.506- could produce dramatically different long-term forecasts. Even small changes in initial conditions can mushroom to enormous proportions.-Superforecasting by Tetlock.

• Oct, 1962: The Cuban Missile Crisis; over 13 terrifying days, the Kennedy administration considered a range of dangerous options to counter the Soviet threat—including outright invasion—before settling on a naval blockade.-Superforecasting by Tetlock.

  • After the fiasco, Kennedy orders an inquiry to figure out how his people could have botched it so badly. It identified cozy unanimity as the key problem and recommended changes to the decision-making process to ensure it could never develop again. Skepticism was the new watchword. Participants were to speak not only as specialists in their area of expertise but as generalists, with a license to question anything. Special counsel Theodore Sorensen and the president’s brother Bobby were designated “intellectual watchdogs,” whose job was to “pursue relentlessly every bone of contention in order to prevent errors arising from too superficial an analysis of the issues.”-Superforecasting by Tetlock.

• Jul, 1962: Soviet Leader Nikita Khrushchev and Cuban Dictator Fidel Castro reach a secret deal to place missiles on Cuba to deter future harassment of Cuba (BBC). By the end of the summer, Cuba was a base for 5000 Soviet soldiers and an array of Soviet intermediate range nuclear missiles that could destroy Washington, DC, and New York City, and the two global superpowers were locked in a crisis that Kennedy estimated, in retrospect, had between a one-third and one-half chance of escalating into nuclear war.-Superforecasting by Tetlock.

• 17 Apr, 1961: Bay of Pigs; a CIA trained Cuban insurgency force is captured at Bay of Pigs in Cuba. When the guerillas land, the 20K strong Cuban Army is waiting for them. Within a few months, nearly the entirety of the 1400 -man force is dead or missing-Superforecasting by Tetlock.

• 1959: Cuban Coup; Fidel Castro becomes Cuban dictator after an armed revolt that overthrows Fulgencio Batista (JFK Library).

• 1954: Psychologist, Paul Meehl, writes a small book that reviews 20 studies showing that well-informed experts predicting outcomes—whether a student would succeed in college or a parolee would be sent back to prison—were not as accurate as simple algorithms that added up objective indicators like ability test scores and records of past conduct.-Superforecasting by Tetlock.

• 1952: Election of Dwight D. Eisenhower as POTUS.-Superforecasting by Tetlock.

  • After World War II, Eisenhower was immensely popular and both parties all but begged him to accept their nominations for the presidency. President Truman even offered to step aside for him. Eisenhower always refused. He sincerely did not want the job. But as the race for the 1952 election shaped up, it became clear that the Republican nomination, and likely the presidency, would be won by an isolationist who promised to bring all American troops home to “the Gibraltar of freedom.” Eisenhower thought this would be a catastrophe that he—and he alone—could stop. “He wanted what was best for his country,” one biographer wrote, “and in the end he decided that he was the best and would have to serve.”-Superforecasting by Tetlock.

• 6 Jun, 1944: The Allies feared that after their troops landed, German tanks would drive them back to the beaches and into the sea, but Hitler had directed that the reserves could only move on his personal command. Hitler slept late. For hours after the Allies landed on the beaches, the dictator’s aides refused to wake him to ask if he wanted to order the tanks into battle.-Superforecasting by Tetlock.

• Mid Feb- Aug, 1942: ~112K Japanese-Americans (2/3 of whom were born in the USA) are shipped to isolated camps ringed with barbed wire and armed guards.-Superforecasting by Tetlock.

• 1941: Creation of the Coordinator of Information (COI), which becomes the Office of Strategic Services (OSS), which later becomes the CIA.-Superforecasting by Tetlock.

• 10 May, 1940: Case Yellow- the German invasion of Belgium and France begins; dozens of German gliders under the command of Lt Rudolf Witzig land in the vicinity of Eben Emael, a massive, largely underground fortress that was the centerpiece in Belgium’s border defense. Most of the gliders landed in fields. Soldiers scrambled out and attacked Belgian troops guarding bridges. Nine gliders landed on the roof of the fort. Soldiers rushed out and destroyed the heavy guns. Eben Emael’s defenders surrendered. Witzig’s own glider had to make an emergency landing in Germany, 100km off target. On top of Eben Emael, a sergeant took charge of the remaining troops and knocked out the Belgian guns. Then another glider swooped down and landed on the fort. Out hopped Rudolf Witzig, who had secured another plane and glider and gotten to Eben Emael only a little behind schedule. As for the other errant glider, the sergeant in charge commandeered two vehicles, drove into Belgium, and improvised a ground attack that netted 121 prisoners.-Superforecasting by Tetlock.

• 19 Jul, 1870- 28 Jan, 1871: The Franco-Prussian War (‘Franco-German War’, ‘War of 1870’) is fought between the Second French Empire and the North German Confederation led by the Kingdom of Prussia. The conflict was caused primarily by Frances’s determination to reassert its dominant position in continental Europe following Prussia’s 1866 victory in their war against Austria. According to some sources, Prussian chancellor Otto von Bismarck deliberately provoked the French into declaring war on Prussia in order to induce the four-independence southern German states- Baden, Württemberg, Bavaria, and Hesse-Darmstadt to join the North German Confederation. The French invaded on 2 August and the German’s invaded France on 4 August. A series of swift Prussian and German victories in E. France culminating in the Siege of Metz and the Battle of Sedan, resulted in the capture of French Emperor Napoleon III and the decisive defeat of the army of the Second Empire. A French Government of National Defense was formed in Paris on 4 Sep which fought on for another five months before German forces defeated it and then besieged Paris for 4 months, taking it on 28 Jan, 1871, effectively ending the war. Near the end of the war, the German states proclaimed their union as the German Empire under Prussian King Wilhelm I and Chancellor Bismarck. An Armistice of France was later signed on 10 May, 1871 as the Treaty of Frankfurt, giving Germany billions of francs in war indemnity as well as most of Alsace and parts of Lorraine. The war saw the rise of Germany as Europe’s continental superpower and the fall of French imperial rule (Wiki).

• 1869: Yasukuni is founded in honor of Japan’s war dead; it now lists ~2.5M soldiers.-Superforecasting by Tetlock.

• 14 Jun- 22 Jul, 1866: The Austro-Prussian War (‘Seven Weeks War’, ‘German Civil War’, ‘Brothers War’, ‘Fraternal War’) is fought between the Austrian Empire and the Kingdom of Prussia, with each being aided by various allies within the German confederation. Prussian forces, allied with Italy and led my Moltke, defeat Austrian forces resulting in Prussian dominance over the German states, the abolition of the German confederation, and the Italian annexation of the Austrian province of Venetia (Wiki).

• 1 Feb- 30 Oct, 1864: The Second Schleswig War (Dano-Prussian War) is fought as the second military conflict over the Schleswig-Holstein territory. The War began when Prussian and Austrian forces crossed the border into the Danish fief Schleswig and ended with the treaty of Vienna which ceded Denmark’s Duchies of Schleswig, Holstein, and Saxe-Lauenburg to Prussia and Austria (Wiki).

• 1761: Posthumous publication of Bayes “An Essay Towards Solving a Problem in the Doctrine of Changes” by his friend Richard Price, which ultimately produces Bayes theorem.-Superforecasting by Tetlock.

  • P(H|D)/P(-H|D) = P(D|H)/P(D|-H) • P(H)/P(-H), Posterior Odds = Likelihood Ratio • Prior Odds

  • In simple terms, the theorem says that your new belief should depend on two things—your prior belief (and all the knowledge that informed it) multiplied by the “diagnostic value” of the new information.

• 25 Aug, 1758: The Battle of Zorndorf; a Prussian force commanded by Prussian King Frederick the Great battles A Russian force commanded by Count William Fermor to a tactical standstill; both armies hold their ground, claiming victory.-Superforecasting by Tetlock.

• 1747: British naval doctor James Lind takes 12 sailors suffering from scurvy, divides them in pairs, and gives each pair a different treatment: vinegar, cider, sulfuric acid, seawater, a bark paste, and citrus fruit.-Superforecasting by Tetlock.

• 1713: Publication of “Ars Conjectandi” by Jakob Bernoulli’s, one of the first books on mathematical probability.-Superforecasting by Tetlock.

___________________________________________________________________________