← Back to book summaries

Summary of "Noise: A Flaw in Human Judgment"

•2 min read
•
psychologyscience

Core Idea

  • Noise (unwanted variability in judgment) costs organizations hundreds of millions and creates injustice—yet remains invisible compared to bias
  • Same case evaluated by equally qualified professionals yields wildly different assessments
  • Reducing noise by systematic structure is as critical as reducing bias

The Problem

  • Humans judge single cases causally, not statistically—missing patterns of inconsistency
  • Hindsight bias and confirmation bias make variability invisible
  • Three noise types exist: level noise (judge severity differences), stable pattern noise (personality-driven), occasion noise (mood/context)

Decision Hygiene Framework

  • Decompose complex judgments into independent dimensions before evaluating
  • Sequence information: withhold biasing context until judgment is made
  • Aggregate independent judgments mechanically (wisdom of crowds reduces noise by ~√n)
  • Use structured scales: anchor decisions in outside view (base rates) and reference cases, not abstract ratings
  • Designate decision observers with bias checklists to catch groupthink

Noise Reduction by Domain

Hiring

  • Break into 4 dimensions: cognitive ability, leadership, culture fit, role knowledge
  • Use structured interviews + work samples (writing tests, code samples beat unstructured conversation)
  • Cap interviews at 4; separate each assessment phase before holistic judgment
  • Replace absolute ratings with relative rankings where possible

Medicine & Diagnosis

  • Deploy diagnostic guidelines (break judgment into independent scored dimensions)
  • Use sequential unmasking: withhold contextual bias until diagnosis rendered
  • Apply structured interview protocols for clinical decisions
  • Implement sentencing guidelines: reduces judge variance by ~40%
  • Use case scales as anchors, not number scales
  • Train judges on frame-of-reference with real examples

Strategic Decisions

  • Estimate-talk-estimate protocol: collect independent forecasts, discuss, re-estimate (avoids cascade bias)
  • Disaggregate complex questions into components before synthesis
  • Delay final intuition until all evidence collected

Performance Ratings

  • Apply frame-of-reference training (case scale anchors, not abstract numbers)
  • Keep dimensions to 3–4, not 11 (prevents halo effect)
  • Use relative rankings where possible

Rules vs. Standards

  • Use rules when: decisions repeat, stakes are high, noise costs exceed implementation costs
  • Use standards when: situations are diverse, values evolve, discretion is needed
  • Monitor rules for perverse incentives; revise if evasion emerges

Action Plan

  1. Conduct a noise audit: have 5+ judges evaluate 10 identical cases; measure variability to quantify baseline
  2. Map decision structure: list independent dimensions for your highest-noise decisions before evaluating cases
  3. Assign independent assessors: separate people to each dimension; minimize communication until aggregation
  4. Anchor in outside view: replace abstract scales with reference cases and base rates
  5. Test cost-benefit: calculate error costs from current noise; if reduction ROI is positive, implement structured process
Copyright 2025, Ran Ding•Privacy•Terms
Summary of "Noise: A Flaw in Human Judgment"