How critical thinking uncovered an error in "Thinking, Fast and Slow"

Craig Cockburn, Edinburgh, Scotland April 2025.

Introduction

Critical thinking is the process of analysing available data, assumptions, and arguments to reach sound conclusions, with deference to looking out for common biases behavioural researchers have spent roughly 60 years documenting. This is not a completely new challenge. Calls for greater application date back to Socrates (C. 470–399 BC) and include Devil’s advocacy (1587), Kriegsspiel (1812) and Red Teaming as applied to both business & military decision making.

Background

Since giving my first conference talks on business decisions in 2019, I’ve been especially drawn to the concept of Red Team Thinking with the aims of offering a clearer framework for understanding and overcoming biases affecting decision making in business. To this end, I’ve drawn from the thoughts of Dave Snowden (Cynefin) and Dr Gary Klein (Premortem) and Bryce Hoffman (who has de-emphasized the existence of separate teams and instead emphasized this as a cognitive exercise within teams, with a special emphasis and the avoidance of arriving at confident predictions too early, if ever P85.

This tendency to jump to conclusions is illustrated by Shane Frederick’s (2005) Cognitive Reflection Test, which was designed to measure how carefully System 2 (reflective reasoning or deliberation) monitors and controls the output of System 1 (reflexive reasoning or Intuition).

Observations

Referencing Professor Frederick’s research, Professor Kahneman averred: “Many thousands of university students have answered the bat-and-ball puzzle, and the results are shocking. More than 50% of students at Harvard, MIT, and Princeton gave the intuitive—incorrect—answer. At less selective universities, the rate of demonstrable failure to check was in excess of 80%.”

Although Professor Kahneman’s summary of Professor Frederick’s research was essentially correct, he somewhat exaggerated Professor Frederick’s “surprising” results.

This exaggeration is hinted at by examining Professor Frederick’s summary of results in the original 2005 paper (see below), which involved 3,428 respondents (most of whom were university students). Source https://pubs.aeaweb.org/doi/pdfplus/10.1257/089533005775196732

Figure 1: The Cognitive Reflection Test (CRT)

1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? ___ cents
2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? ___ minutes
3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? ___ days

Table 1: CRT Scores, by Location

In this data, the highest performance was from students at MIT where “only” 48% got a perfect score (3/3). The remainder of the table reveals that 30% of MIT students missed one of the three questions, 16% missed two of them and 7% missed all three.

It is not true that most people missed the bat & ball problem. When I asked him about the seeming discrepancy between the claim in Thinking Fast & Slow and his actual source data, Professor Frederick responded: “Hi Craig. Good detective work. You are correct.”

This correspondence followed.

CRAIG: The problem is that the book claims it was the bat and ball question specifically, however the referenced data does not state this level of detail.

SHANE: Correct, I did not report item level results in my 2005 paper (Table 1 above). But that does set bounds on what item level results are possible, as you rightly note. Further, I do “of course” have that data. And I looked it up to give you the correct answer.

CRAIG: It looks like the book text is over-simplifying what the data says. … Surprisingly I couldn’t find any comments online regarding this point as people seem to be citing the book rather than the original study.

SHANE: I would not say that this is oversimplifying what Table 1 reveals but misstating it.

In any case, the solution rates for the “Bat & Ball” item for the 3 mentioned prestigious schools were…

MIT: ……….59% (N= 61)

Princeton: 52% (N= 121)

Harvard:… 49% (N= 51)

(In other words, at those three schools, the problem was missed by 41%, 48% & 51%, respectively.)

CRAIG: Were you able to discuss this with the late Dr Kahneman at all when he quoted your work?

SHANE: Danny & I were good friends and in frequent correspondence. I’m a huge fan of Danny’s and he was a great mentor for me & a great champion of mine. FWIW, I believe I did mention it to him, yes, and could probably even find the correspondence, but it doesn’t seem very important. I don’t regard this as malfeasance, just slight carelessness. We all make mistakes. Moreover, the qualitative conclusion (that even at the most prestigious institutions, MANY people make the mistake) IS certainly correct. But, you are right; it’s not quite as many as his summary implied. Further, regarding the accuracy of the quote in question, I should note that although my Total sample in the 2005 paper was 3,428, “only” about 2500 of those respondents were university students and only a small fraction of those were from “prestigious universities.”* The three “prestigious” samples were, in fact, small convenience samples. (I did my post doc at Princeton with Danny, was teaching at MIT, and Harvard was nearby.)

*I’m coding University of Michigan & CMU as NOT prestigious. The samples from those two schools were moderately large. (N= 1267 at University of Michigan & N=746 at Carnegie Mellon University)

Thus, returning the quote in question (below), what Professor Kahneman should have written was: “When the bat & ball problem was given to college students, most missed it. Indeed, even at prestigious universities (e.g., Harvard, MIT, Princeton) roughly half missed it. At less selective universities (e.g. the University of Toledo), the error rate exceeds 80%.”**

** The 2nd part of Kahneman’s composite claim is correct. For example, of the 138 students given the problem at University of Toledo, only 17% responded correctly (5 cents).

I hope you have found this updated information useful, and it also means that students at MIT are shown to reflect more than originally indicated in the book!

Shane May 22, 2025: Just to re-emphasize, Kahneman’s’ claim in TF&S was that the error rate exceeded 50% at three of the most prestigious universities. In fact, the error rates for this problem were 41%, 48%, & 51% at these three universities, respectively (and somewhat less than 50%, overall). Thus, while TF&S did slightly exaggerate my results, this is a minor error – especially considering the many more spectacularly false claims that pollute my field – and nearly every area of science, see, e.g., (Meyer, et al., 2015).

That said, I remain impressed by anyone who is a sufficiently careful reader to recognize, inquire about, call attention to, & correct even a (slightly) erroneous claim.

Best regards, Shane

Summary

Critical thinking helps you to uncover facts, discern the truth and supports rational debate, constructive challenge and clearer decision making. This is true even when challenging the work of Nobel Laureates who, being human, can make mistakes just like the rest of us. Although the example here is numerical, the principles are still just as valid for non-numerical problems and this is where the majority of the work falls in a business context.

According to the world economic forum, 2025 report, “Analytical thinking remains the most sought-after core skill among employers, with seven out of 10 companies considering it as essential in 2025.” https://www.weforum.org/publications/the-future-of-jobs-report-2025/in-full/2-jobs-outlook/#2-jobs-outlook

A key principle of critical thinking that it is with a constructive intent.

“‘Tis not enough, taste, judgment, learning, join; * In all you speak, let truth and candour shine:* * That not alone what to your sense is due,* * All may allow; but seek your friendship too.”*

– Alexander Pope, 1711.

Further Reading

See (Andrew Meyer S. F., 2023) and (Andrew Meyer Y. A.-H., 2024) (one of the last joint papers of Daniel Kahneman before his death in 2024)

Acknowledgements

With thanks to Shane Frederick, Professor of Marketing at Yale School of Management for collaborating on this article and supplying me with the relevant source data. Professor Frederick co-published with Dr Kahneman between 2002 and 2024 in the field of decision sciences.

https://faculty.som.yale.edu/shanefrederick/

and to Dr Daniel Kahneman, Nobel Prize Laureate, for revolutionising how we think about thinking.

Author Biographies

Craig Cockburn, B.Sc., M.Sc. CITP, CEng, FBCS has worked as a business consultant since 2007 helping businesses work more effectively and efficiently. In 2018 he was confirmed as a Chartered Fellow of the Chartered Management Institute. His conference speaking topics include critical thinking, strategy, decision support in business, teaching practices for neurodiversity and remote working. He is credited with inventing an early browser (1990), publishing on remote working (1993) and on the acknowledgements for HTML5. His Google scholar profile also includes his undergraduate work on AI and references to his vocational work on Scottish traditional culture.

Shane Frederick. Professor Frederick’s research focuses on preference elicitation, framing effects, intertemporal choice, and decision-making under uncertainty. Before coming to Yale SOM, he was associate professor of management science at Sloan School of Management at MIT. Prior to MIT, he was a research associate and lecturer at the Woodrow Wilson School of Public & International Affairs at Princeton University. Professor Frederick holds a PhD in decision sciences from Carnegie Mellon University, an MA in resource management from Simon Fraser University, and a BS in zoology from the University of Wisconsin.

Contact details

Craig Cockburn https://orcid.org/0009-0006-9995-8797 craig@siliconglen.com https://www.craigcockburn.com and https://www.redteamthinking.com

Shane Frederick https://orcid.org/0000-0003-0367-0678

References

Andrew Meyer, S. F. (2023). The formation and revision of intuitions. Cognition, 240. doi:10.1257/0895330005775196732

Andrew Meyer, Y. A.-H. (2024). Cognitive reflection is a distinct and measurable trait. PNAS. doi:10.1073/pnas.2409191121

Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 25–42. doi:10.1257/089533005775196732

Hoffman, B. G. (2017). Red teaming: How your business can conquer the competition by challenging everything. New York: Crown Business.

Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.

Meyer, A., Frederick, S., Burnham, T. C., Pinto, J. D., Boyer, T. W., Ball, L. J., . . . Schuldt, J. P. (2015). Disfluent fonts don’t help people solve math problems. Journal of Experimental Psychology: General, 144(2), e16–e30. doi:10.1037/xge0000049

Keywords: #criticalthinking #cognitivereflection #redteamthinking #system2

Get in touch now: Contact me