# Cognitive Reflection Test

The **Cognitive Reflection Test** (**CRT**) is described by some to be the world’s shortest intelligence test, consisting of only **3** questions. Due to its popularity and some people’s familiarity with the questions, other lesser-known items have been added to revised versions of the test.

- Questions:
**3** - Duration:
**3–15 minutes** - Type:
**intelligence test** - Author:
**Shane Frederick** - Publishing year:
**2005** - Seminal Paper:
**Cognitive Reflection and Decision Making (Frederick, 2005)**

Take the test here:

### CRT

**Note:** Please take the test before reading through the information.

Additionally, if you have dyscalculia and/or dyslexia, this could result in a lower score (read more in the **Scoring** section).

**Who the test is designed for**

**Adults**(age**16+**)

## What it tests

- The CRT was not actually created as a measure of intelligence, but as a simple measure of one type of cognitive ability—the ability or disposition to reflect on a question and resist reporting the first response that comes to mind.
^{[1]}Cognitive Reflection and Decision Making (Frederick, 2005) - The CRT thus seems to correlate with
**response inhibition**, one of the**executive functions**. - CRT scores are predictive of the types of choices that play a large role in tests of decision-making theories, such as
**expected utility theory**and**prospect theory**.^{[2]}Cognitive Reflection and Decision Making (Frederick, 2005) - But the CRT does have a moderate correlation with measures of intelligence, such as the
**IQ**test. - The CRT also correlates highly with various measures of
**mental heuristics**,^{[3]}Cognitive abilities and behavioral biases (Oechssler, Roider & Schmitz, 2009) which are strategies to quickly form judgments, make decisions, and find solutions to complex problems. - Test scores are also significantly related to subjects’ time and risk preferences.
^{[4]}Cognitive abilities and behavioral biases (Oechssler, Roider & Schmitz, 2009)

**Versions & translations**

- The CRT has been translated into different languages.

**Taking the test**

The original CRT consists of just **3** questions, which requires logic and calculus to answer. Answers will consist of numbers. Take the test below!

(The correct answers to the questions can be found in the **Answers** section)

## Cognitive Reflection Test

- A bat and a ball cost
**$1.10**in total. The bat costs**$1.00**more than the ball.

How much does the ball cost?

_____ cents. - If it takes
**5**machines**5**minutes to make**5**widgets,

how long would it take**100**machines to make**100**widgets?

_____ minutes. - In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes
**48**days for the patch to cover the entire lake,

how long would it take for the patch to cover half of the lake?

_____ days.

## Scoring

- Scoring range:
**0–3** - Threshold score:
**2** **83%**of people missed at least one of the questions.- NB: If you score
**0**, but you have an average to high IQ, this test could indicate**dyscalculia**.^{[5]}Cognitive reflection vs. calculation in decision making (Sinayev & Peters, 2015)

The CRT is only available as a self-scoring test.

In his book **David and Goliath**, **Malcolm Gladwell** says the following about the average scores on the CRT:

Frederick gave the CRT to students at nine American colleges, and the results track pretty closely with how students from those colleges would rank on more traditional intelligence tests. Students from the Massachusetts Institute of Technology—perhaps the brainiest college in the world—averaged

2.18correct answers out of three. Students at Carnegie Mellon University in Pittsburgh, another extraordinarily elite institution, averaged1.51right answers out of three. Harvard students scored1.43; the University of Michigan, Ann Arbor,1.18; and the University of Toledo0.57.

## Validity

##### How reliable, accurate, valid, and up to date is the test?

- The original CRT is an attractive test as it involved only the first three items. Despite requiring only a few minutes to administer, the CRT’s predictive validity equals or exceeds other cognitive tests.
- The CRT is the world’s shortest scientifically validated test. However, an estimated
**44–51%**of the people who take the test are already familiar with the questions.^{[6]}Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success? (Haigh, 2016)^{[7]}A limitation of the Cognitive Reflection Test: familiarity (Stieger & Reips, 2016) As such, researchers have developed revised versions, such as the**CRT-Long**,^{[8]}The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT) (Primi et al., 2015) the**CRT-7**,^{[9]}Assessing miserly information processing: An expansion of the Cognitive Reflection Test (Toplak, West & Stanovich, 2013) and the**CRT-2**,^{[10]}Investigating an alternate form of the cognitive reflection test (Thomson & Oppenheimer, 2016) which feature different questions you may not yet be familiar with. - People who have seen the CRT tend to outscore those who have not. And yet research shows that the CRT remains robust despite multiple exposures.
^{[11]}The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT) (Primi et al., 2015) - There is, however, a strong male bias in the test. Shane Frederick already showed that men perform better in the CRT,
^{[12]}Cognitive Reflection and Decision Making (Frederick, 2005) and many other studies have shown the same. A study based on**44,558**people (**52.76%**of whom were female) shows that:^{[13]}Cognitive reflection test: Whom, how, when (Brañas-Garzaa, Kujal & Lenkei, 2015)- Men perform better in every single question;
- Women are more likely to answer none of the questions correctly; and
- Men are more likely to answer all three questions correctly.

- A study on the gender differences (based on the longer version of the test, the
**CRT-L**) shows that the difference in performance between men and women is due to differences in**math anxiety**(tension and anxiety around working with numbers) and**mathematical reasoning**. So cognitive reflection is influenced by**numerical skills**and related feelings like anxiety.^{[14]}Are there gender differences in cognitive reflection? Invariance and differences related to mathematics (Primi et al., 2017) Since the**CRT-2**relies less on numeracy,^{[15]}Investigating an alternate form of the cognitive reflection test (Thomson & Oppenheimer, 2016) for women that is probably a more accurate measure of the test than the CRT.

## Discussion

##### Natalie:

The things to keep in mind are:

- Has the person seen the test before (the ball/bat question is often shared on social media).
- Men score higher than women.
- A person with dyscalculia will be at a disadvantage with this test even if they are bright.
- The CRT was thought to measure the ability to recognize and override errors that are made on quick and effortless intuition and instead use a system that is more controlled and slow. However, CRT items require math ability to be answered correctly and research has shown that mathematical ability is more important than cognitive reflection.

##### Kendall:

On seeing three statements, each involving numbers, I knew the CRT would be a problem. A score of **0** would be disheartening if I were unaware of my dyscalculia. Thus it is highly relevant to understand this when you are considering the results.

##### Martin:

I imagine knowing what the CRT actually measures can greatly influence your performance on the test. I remember that when I first heard the bat-and-ball question, I got the answer wrong. So when I took the CRT, I was careful to think it through rather than offering an intuitive response. If you take the test thinking it’s an intelligence test, you will be more likely to give the intuitive and therefore incorrect answer.

On the other hand, not knowing what the test measures may give a more accurate measure of your propensity to override an intuitive—but incorrect—response with a more analytical correct response. I think it will give a fair indication of your intelligence irrespective of whether you understand what the test measures and correlates with though. But if you want to take this test as a measure of intelligence and you have challenges with numbers, do take the **CRT-2** instead.

**Answers**

- Correct answer =
**5 cents**; intuitive answer =**10 cents**. - Correct answer =
**5 minutes**; intuitive answer =**100 minutes**. - Correct answer =
**47 days**; intuitive answer =**24 days**.

## Explanations

**1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? (Answer: 5 cents.)**

Note that the bat costs **$1.00** MORE than the ball—whatever the price of the ball may be. So the bat doesn’t cost **$1.00**, and the ball **$1**, as that would mean the bat costs **$0.90** more than the ball. The ball should cost **$0.05** in order for the bat to cost **$1.00** more (= **$1.05**), for a total of **$1.10**.

If you are more mathematically inclined, let’s say the ball costs **X**. We know that the bat costs **$1** more than **X**, meaning that its price is **X + 1**. The total price of the bat and ball, then, is **X + 1 + X**. That’s a total of **2X + 1**, which we already know is **1.1** (**$1.10**). In that case, we know that **2X = 0.1**, which means that **X = 0.05**.

**2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? (Answer: 5 minutes.)**

Each machine makes a widget in **5** minutes. The number of machines has no effect on how long it takes, because it will always take **5** minutes. So **100** machines working together will produce **100** widgets in **5** minutes.

Note also that the question can be interpreted such that at least **5** machines are required to work in unison to produce **5** widgets, but this also has no bearing on the answer.

**3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? (Answer: 47 days)**

Every day the patch doubles in size, so every PREVIOUS day the patch was half the size. So if the patch takes **48** days to cover the lake, then the before-last day, on day **47**, the patch covers half the lake.

## Recommended next steps

After the CRT, the tests below are suggested.

**Autism Spectrum Quotient (AQ)**

*A simple screening test that is used as a basis for pursuing a formal autism evaluation*

**Empathy Quotient (EQ)**

*Measures Theory of Mind*

**Systemizing Quotient (SQ)**

*Measures systemizing and pattern finding*

Online autism tests can play an essential role in self-discovery and may inform your decision to pursue a formal diagnosis. For a formal assessment, please see a knowledgeable professional trained in assessing autism.

### References

Dyscalculia, Executive functions, Inhibitory control, Intelligence, Intelligence quotient (IQ), Intuition, Mathematical anxiety, Mathematical reasoning, Mathematics, Psychometric test, Response inhibition