Essays北极甜虾 · No. 19

同一道题,换张脸就会了

学不会,
是概念太难,
还是讲法错了?

贝叶斯、四张卡片和洛书
有些概念,我们习惯说“人脑天生不擅长”。可是同一道题换一种写法,答对的人能从一成多变成快一半。这篇用两个心理学里的老实验,加上一个能和朋友玩的小游戏,聊聊一个问题:学不会的时候,到底该怪概念,还是怪讲法。
课本的写法患病率 1%病人查出阳性 90%没病误报阳性 10%查出阳性,真的有病的概率?直觉:九成左右想成 1000 个人阳性、真病:9阳性、没病:99答案:9 ÷ 108 ≈ 8.3%
图 1同一道题的两张脸。左边是课本写法,右边把它画成 1000 个人:红点是查出阳性、真的有病的 9 个人,深灰点是查出阳性、其实没病的 99 个人。

先请你做一道题,别急着往下翻。

某种病,人群里的患病率是 1%。有一种检测,病人去查,90% 会显示阳性;没病的人去查,也有 10% 会误报阳性。现在你查出来是阳性。你真的得病的概率有多大?

想好了吗?很多人的第一反应是九成左右,毕竟检测能查出 90% 的病人。

换个说法再来一遍。想象 1000 个人去做检测。其中 10 个人有病,查出阳性的是 9 个。剩下 990 个人没病,误报阳性的是 99 个。那么所有查出阳性的人里,真有病的占多少?

这回大概不用算多久:阳性一共 9 + 99 = 108 人,真病的只有 9 个,约 8.3%。

两段话说的是同一件事,数字一个没改。可第一种写法让人往九成猜,第二种写法让人几乎一眼看出答案。

这篇想聊的就是这个现象背后的问题:一个学生学不会某个概念,究竟是概念本身太难,还是我们挑错了讲它的方式和顺序?

两种解释,预测不一样

“贝叶斯推理违背人类直觉”,这句话你多半听过。它背后的意思是,人脑天生不是按概率论设计的,学不会怪不得谁。

另一种解释是:人脑能做这种推理,只是我们一直用一种别扭的格式把题目喂给它。

这两种解释听起来像在抬杠,其实能用实验分出高下。如果难在概念本身,换一种写法,成绩不该有太大变化;如果难在写法,换一下就该有明显变化。题目开头那两段话,就是一个小小的实验。下面看看心理学家做过的大实验。

贝叶斯:换成人数

先说坏消息。据 1978 年发表在《新英格兰医学杂志》上的一项调查,研究者拿一道类似的题去问哈佛医学院的学生和医生,60 个人里只有 11 个答对。最常见的错误答案是 95%,比正确答案高了几十倍。另据 Eddy 1982 年的报告,100 位医生估算乳腺 X 光阳性后真患乳腺癌的概率,95 位给出 70% 到 80%,按贝叶斯公式算只有 7.5% 左右。

这类结果攒多了,就有了“人脑不是贝叶斯机器”的说法。

1995 年,Gerd Gigerenzer 和 Ulrich Hoffrage 做了一件很简单的事。他们找来萨尔茨堡大学 60 个没学过贝叶斯公式的学生,做 15 道这样的题。同一道题准备两个版本:一个用概率写(“患病率 1%,阳性率 80%”),一个用人数写(“每 1000 人里 10 人患病,其中 8 人阳性”)。他们管后一种叫自然频数。

结果,概率版里能按贝叶斯思路做对的只有 16%;人数版里是 46%。这个比例的口径还很严:不光答案要对,写下来的草稿也要确实是在用贝叶斯的思路算,蒙对的不算。

后来他们把同样的办法拿去问医生。1998 年的研究里,48 位慕尼黑和杜塞尔多夫的医生做四道诊断题,用概率写时,只有 10% 的估算是对的;换成人数,升到 46%。

1000人1%99%10有病990没病9阳性1阴性99阳性891阴性90%10%
图 2把开头那道题画成一棵树。从 1000 个人出发,先按有病没病分,再按阳性阴性分。答案就是两个红框里的人数之比:9 / (9 + 99)。

为什么换成人数就好做?两种写法数学上完全等价,可算起来不等价。

Gigerenzer 他们打过一个比方:同一个数,可以写成阿拉伯数字,也可以写成罗马数字,信息一模一样。但你试试用罗马数字做长除法。一台只认阿拉伯数字的计算器,喂给它罗马数字,它算出一堆乱码,不能说明它不会乘法。

概率写法要你在脑子里做三次乘法,再做一次加法和一次除法,中间还得记住哪个百分比是对谁说的。人数写法把这些乘法提前替你做完了,你只需要数人头。认知科学家 Larkin 和 Simon 1987 年有一篇名文,标题就叫《为什么一张图有时胜过一万个字》,讲的是同一个道理:两种表示装着一样多的信息,但一种把答案摆在明面上,另一种要你自己挖出来。

证据值多少,要看两头

人数写法还顺带纠正了一个更深的直觉错误。我们评价一条证据,习惯只看一头:病人里有多少人阳性?90%,很高,所以阳性很说明问题。

可证据值多少,还取决于另一头:没病的人里有多少人也阳性?如果没病的人也经常阳性,这条证据就没那么值钱。

举个生活里的例子。“这个人经常熬夜”,能不能说明他是程序员?光知道程序员爱熬夜不够,还得问:不是程序员的人熬不熬夜?大学生、医生、追剧的人都熬夜,这条证据就很弱。

这就是贝叶斯推理真正要养成的习惯:评价一条证据,永远拿它在两种解释下各有多常见来比。公式只是把这个习惯写成了算式。

数学(可跳过)贝叶斯公式有一个很好记的“赔率”形式。记 为“有病”, 为“阳性”:

左边是看到证据以后的赔率,右边第一项是原来的赔率,第二项叫似然比,就是上面说的“两头一比”。证明只要一行:把 和 相除, 约掉。

代入开头那道题:原来的赔率是 10 : 990,似然比是 90% ÷ 10% = 9,乘起来是 90 : 990,也就是 1 : 11。赔率 1 : 11 换成概率是 1/12,约 8.3%,和数人头的结果一样。∎

原来:病人对没病990 没病10阳性:病人一侧 × 90%,没病一侧 × 10%,相当于病人一侧放大 9 倍看到阳性以后990 没病9090 : 990 = 1 : 11,概率 1/12 ≈ 8.3%
图 3赔率形式。原来是 10 个病人对 990 个健康人;阳性这条证据让病人一侧放大 9 倍,变成 90 对 990,也就是 1 对 11。证据很强,可病太少见了。

四张卡片:换成规矩

第二个实验更有名。再做一道题。

桌上有四张卡片,每张一面是字母,另一面是数字。你看到的四面是:A、K、4、7。有一条规则:“如果一面是元音字母,那么另一面是偶数。”要检查这条规则有没有被违反,最少要翻哪几张?

大多数人会翻 A 和 4。正确答案是 A 和 7。

翻 A 没问题:它背面要是奇数,规则就破了。翻 4 没用:规则只说元音背后要是偶数,没说偶数背后必须是元音,4 背后是什么字母都不违规。真正该翻的是 7:它背面要是元音,规则就破了。

这道题是英国心理学家 Peter Wason 1966 年提出的,叫沃森选择任务。按 Sperber 等人 1995 年的总结,这道标准的抽象版,通常只有一成左右的人选对 A 和 7。

再做一道。你是酒吧的检查员,规则是“如果一个人在喝酒,那么他必须年满 18 岁”。你面前有四个人,每人只知道一条信息:一个在喝啤酒,一个在喝果汁,一个 16 岁,一个 22 岁。你要查谁?

这题几乎不用想:查喝啤酒的那位(看他成年没有),查 16 岁的那位(看他喝的是不是酒)。

如果一面是元音,另一面就是偶数A元音K辅音4偶数7奇数如果在喝酒,就必须年满 18 岁啤酒在喝酒果汁没喝酒22 岁成年16 岁未成年
图 4两道题一一对应:A 对喝啤酒,K 对喝果汁,4 对 22 岁,7 对 16 岁。要翻的都是红色那两张。

两道题的逻辑结构一模一样,难度却天差地别。据 Griggs 和 Cox 1982 年的实验,换成饮酒年龄的版本后,约 74% 的佛罗里达大学生答对。

数学(可跳过)记 为“一面是元音”, 为“另一面是偶数”,规则是 。这条规则唯一的反例是“ 成立而 不成立”:一面元音、一面奇数。

所以要找的是可能藏着这种反例的卡片。看得见 的(A),要看它背面是不是 ;看得见 的(7),要看它背面是不是 。另外两张,无论背面是什么都凑不出反例。

换句话说, 和它的逆否命题 等价,但和 不等价。选 4 的人,其实是把规则当成了 。∎

你可能已经注意到了,两个实验里的错误长得很像。

贝叶斯四张卡片抽象写法条件概率如果……那么……常见的错把 P(阳性|病)当成 P(病|阳性)把 P → Q当成 Q → P换一张脸数人头查酒吧要养成的习惯证据两头比专找反例
图 5两种错误都是把方向弄反。

“病人里多半阳性”被当成“阳性的人多半有病”;“元音背后是偶数”被当成“偶数背后是元音”。两种错,都是把一个有方向的关系读反了。换一张脸以后,方向一下子就清楚了。

帮忙的不是“具体”

读到这里,很容易得出一个简单的结论:抽象的东西难,换成具体的、生活化的就好了。素材里也是这么说的。我起初也这么想,但查完资料,觉得这个结论只对了一半。

先看贝叶斯。人数写法确实有帮助,但没有神奇到那个地步。据 2017 年一项汇总了大量研究的元分析,用概率写,平均只有 4% 的人做对;换成人数,平均也只到 24%。也就是说,拿到人数写法,还是有四分之三的人做错。2018 年的一项研究发现了一个有意思的原因:很多人拿到人数以后,又把它换回百分比去算,绕了一圈回到了老路上。

再看卡片。具体的故事并不总是有用。Manktelow 和 Evans 1979 年做了五个实验,把字母数字换成各种生活内容,成绩都没怎么提高。Griggs 和 Cox 自己也发现,换成大学生不熟悉的规则(比如某个邮政规定),也没有帮助。

反过来,不具体的写法也能帮忙。据 Cheng 和 Holyoak 1985 年的报告,同样是抽象规则,写成“如果要做某件事,就必须先满足某个条件”这种“许可”的样子,答对率是 61%;写成任意的字母数字规则,只有 19%。规则里一个具体的东西都没有,成绩还是翻了三倍。

0%20%40%60%80%贝叶斯:学生Gigerenzer 等 1995概率 16%人数 46%贝叶斯:医生Hoffrage 等 1998概率 10%人数 46%贝叶斯:元分析McDowell 等 2017概率 4%人数 24%卡片:许可规则Cheng 等 1985任意规则 19%许可规则 61%
图 6几项研究里,换一种写法前后的正确率。各研究的题目、人群和计分口径都不同,只能各自看前后对比,不能横着比高低。数据取自原文或其转述,见文末参考资料。

心理学家对“为什么换了写法就好做”争了几十年,到现在也没吵出结论。有人说人类进化出了专门“抓骗子”的本能(Cosmides 1989);有人说我们在生活里攒下了“许可”“义务”这类推理套路(Cheng 和 Holyoak);还有人说人数写法起作用,是因为它把“阳性的人里有多少真病”这层包含关系摆到了明面上(Barbey 和 Sloman 2007)。

这些解释细节不同,却指向同一个方向:帮上忙的不是“具体”本身,是新的写法让题目的结构现了形,正好对上了你脑子里本来就会的东西。具体例子能做到这一点,就有用;做不到,就白搭,甚至碍事。

“碍事”不是夸张。2008 年《科学》杂志上有一项研究,让学生通过量杯、披萨这类具体例子学一个数学概念,另一组用抽象的符号学。学完以后换到新情境里,抽象组考得更好。具体的细节太抢眼,学生记住了量杯,没记住结构。不过要说清楚,这项研究后来有两次重复都没能得出同样结果,现在还有争议。

同一个游戏,三张脸

讲到这里,给你一个能亲手试的例子。

小游戏:凑 15(先玩,再往下看)
  1. 找个朋友,桌上摆 1 到 9 九个数字。
  2. 两人轮流拿走一个数字,拿走了就归自己。
  3. 谁先在自己手里凑出三个数,加起来正好等于 15,谁就赢。九个数都拿完了还没人凑出来,算平局。

玩两局你会发现,这个游戏挺费脑子。每拿一个数,都得盘算自己手里哪两个数还差多少,对手又快凑成了哪一组。

现在揭晓:这个游戏其实就是井字棋。

井字棋洛书凑 15492357816你拿到258朋友拿到472 + 5 + 8 = 15,同一条线和为 15 的三个数,一共只有 8 组:1 5 91 6 82 4 92 5 82 6 73 4 83 5 74 5 6
图 7同一个游戏的三张脸。把 1 到 9 按洛书的样子排成九宫格,每一行、每一列、两条对角线加起来都是 15,而且加起来是 15 的三个数,全都在一条线上。凑 15,就是在这张棋盘上下井字棋。

把 1 到 9 排成上面这个九宫格,就是古人说的洛书,也是最小的幻方。它有一个性质:三个不同的数加起来等于 15,当且仅当它们在格子里连成一条线。所以“拿数字”就是“下棋子”,“凑 15”就是“三子连线”。

数学(可跳过)从 1 到 9 里挑三个不同的数,和为 15 的组合一共只有 8 组:

{1,5,9},{1,6,8},{2,4,9},{2,5,8},{2,6,7},{3,4,8},{3,5,7},{4,5,6}。

洛书的 3 行、3 列、2 条对角线正好也是 8 条线,逐一对照,恰好就是这 8 组,一组不多,一组不少。还能顺便看出谁最值钱:5 出现在 4 组里,对应棋盘中心;2、4、6、8 各出现 3 次,对应四个角;1、3、7、9 各出现 2 次,对应四条边。下井字棋先占中心,玩凑 15 就该先拿 5。∎

同一个游戏,画成格子,小孩子都会下,还都知道先占中间;写成数字,大人也得掰着手指头算。游戏规则、输赢、最优策略,一样都没变,变的只是它长什么样。

心理学家很早就拿这类“同构题”做过实验。最有名的是汉诺塔:Kotovsky、Hayes 和 Simon 1985 年把汉诺塔改写成“外星怪兽换球”之类的故事,结构完全一样,有的版本却难得多。

顺序:先长直觉,再给名字

表示之外,还有顺序的问题。

我们最熟悉的教法是:先给定义,再讲定理,然后例题,最后应用。贝叶斯就是先写出公式,再拿体检题套。

另一条路是反过来走:先给一个真问题,让你凭直觉猜,猜错了;然后换一种写法,让你自己看出规律;规律站稳了,再告诉你它叫什么、怎么写成公式;最后换几个完全不同的场景,让你认出它。

常见的顺序公式定理例题应用另一条路真问题猜错换写法看出规律公式换场景公式出场的位置不同:一个在开头,一个在你已经懂了以后
图 8两种教学顺序。上面那条,公式第一个出场;下面那条,公式出场时,你已经知道它在说什么了。

按第二条路,公式不再是一串要背的符号,而是在描述一件你已经懂的事。之前聊过“先给问题、后讲解”的研究,这里不重复。值得补一句的是,教育研究里有个说法叫“具体逐渐淡出”(concreteness fading):从具体的东西开始,一步步换成更抽象的写法,最后把具体的拐杖拿掉。前面说过,具体例子有时会让人只记住表面,最后那一步“拿掉拐杖”,就是为了防这个。

三把尺子

说一种教法“好”,要先说清楚拿什么量。至少有三把尺子:

一是当场:刚学完,能不能做对?二是保持:一周、一个月以后,还会不会?三是迁移:题目从体检换成法庭证据、科学实验,还能不能认出这是同一回事?

这三把尺子量出来的结果,常常不一样。

2001 年,Peter Sedlmeier 和 Gigerenzer 写了一个电脑教程,标题很直白,叫《两小时内教会贝叶斯推理》。他们比较了两种教法:一种教学生把概率代进贝叶斯公式(“套公式”),一种教学生把题目画成人数树或者人数格子(“换写法”)。教完当场测一次,一周后测一次,五周后再测一次。

0%25%50%75%100%训练前刚学完一周后五周后套公式:20%30%60%频数树 90%频数格子 75%100%每组十几人,10 道题,答对题数的中位数
图 9芝加哥大学学生,每次测 10 道题,纵轴是答对题数的中位数(百分比)。套公式那组,五周后掉回了 20%;画人数的两组没有掉。每组开始时 14 到 20 人,后两次测验有人缺席。数据读自 Sedlmeier 和 Gigerenzer 2001 年论文的图 9(宽松计分)。

当场那把尺子,两种教法都很有效:套公式组从 0 升到 60%,画人数的两组升到 75% 和 90%。

保持那把尺子,差别就出来了。套公式组五周后掉回 20%,差不多是没学过的样子;画人数的组,五周后一点没掉,有一组还更高了。

要老实说一句,芝加哥这一轮后来缺席的人不少,五周后频数树那组只剩 5 个人,单看这一轮不够硬。所以他们在柏林又做了一遍,56 个人一个没少:套公式组刚学完 64%,五周后 43%;画人数树的组刚学完 86%,五周后还是 86%。

迁移那把尺子,结果出乎我意料:两种教法都迁移得很好,换成没见过的新题,成绩和练过的题差不多。素材里有一句“学会套公式的学生,可能没学到能迁移的思维方式”,在这项研究里没有得到支持。

不过这里的“迁移”是近迁移,新题仍然是标准的贝叶斯文字题。从医学检测迁到法庭证据、迁到日常判断这种远迁移,我没找到认真测过的研究。这可能才是最值得做的实验。

直觉能被造出来吗

素材最后提了一个问题:数学直觉能不能被系统地构造出来,而不是靠大量练习碰运气碰出来?

我的回答是:能造出一部分,而且最硬的证据在“记得住”上。

换一种写法,能让一个概念从“只有少数人会”变成“一大批人会”,五周以后还记得。这说明很多我们以为的“天生学不会”,其实是“没用对写法”。在这一点上,素材的判断是对的:贝叶斯推理没有那么反直觉,是我们一直在用一种反直觉的方式教它。

但也不能说过头。最好的写法也只让一半左右的人做对,元分析的平均数更低;具体例子有时帮忙,有时碍事;而“认出两道题其实是同一道”这一步,到现在也没有谁找到稳定的教法。概念本身的难度,是存在的,只是比我们以为的小。

如果要我给教数学的人一条建议,那就是:别只教一张脸。同一个结构,至少给学生看两三张不同的脸,让他们自己发现底下是同一个东西。体检和熬夜,元音和啤酒,井字棋和凑 15。等哪天他在一个从没见过的地方,自己认出了那个熟悉的结构,那才算是真的学会了。

写在最后

下次你,或者你的孩子,对着一个概念怎么都学不会的时候,先别急着下结论说“我不是学这个的料”。

也许你缺的不是脑子,而是另一张脸。

很多时候,
难的不是概念,是它的长相。
留言区聊聊

你有没有过这种经历:一个概念学了好久不懂,换个讲法突然就懂了?是哪个概念,那个讲法又是什么?

参考资料
  1. G. Gigerenzer, U. Hoffrage (1995), How to improve Bayesian reasoning without instruction: Frequency formats, Psychological Review 102(4): 684–704
  2. U. Hoffrage, G. Gigerenzer (1998), Using natural frequencies to improve diagnostic inferences, Academic Medicine 73(5): 538–540
  3. P. Sedlmeier, G. Gigerenzer (2001), Teaching Bayesian reasoning in less than two hours, Journal of Experimental Psychology: General 130(3): 380–400
  4. W. Casscells, A. Schoenberger, T. B. Graboys (1978), Interpretation by physicians of clinical laboratory results, New England Journal of Medicine 299: 999–1001
  1. D. M. Eddy (1982), Probabilistic reasoning in clinical medicine: Problems and opportunities,收于 Judgment under Uncertainty: Heuristics and Biases(剑桥大学出版社)
  2. M. McDowell, P. Jacobs (2017), Meta-analysis of the effect of natural frequencies on Bayesian reasoning, Psychological Bulletin 143
  3. P. Weber, K. Binder, S. Krauss (2018), Why can only 24% solve Bayesian reasoning problems in natural frequencies, Frontiers in Psychology 9: 1833
  1. A. K. Barbey, S. A. Sloman (2007), Base-rate respect: From ecological rationality to dual processes, Behavioral and Brain Sciences 30(3)
  2. J. H. Larkin, H. A. Simon (1987), Why a diagram is (sometimes) worth ten thousand words, Cognitive Science 11: 65–99
  3. P. C. Wason (1968), Reasoning about a rule, Quarterly Journal of Experimental Psychology 20(3): 273–281
  4. D. Sperber, F. Cara, V. Girotto (1995), Relevance theory explains the selection task, Cognition 57: 31–95
  1. R. A. Griggs, J. R. Cox (1982), The elusive thematic-materials effect in Wason's selection task, British Journal of Psychology 73: 407–420
  2. K. I. Manktelow, J. St. B. T. Evans (1979), Facilitation of reasoning by realism: Effect or non-effect?, British Journal of Psychology 70: 477–488
  3. P. W. Cheng, K. J. Holyoak (1985), Pragmatic reasoning schemas, Cognitive Psychology 17(4): 391–416
  1. L. Cosmides (1989), The logic of social exchange, Cognition 31: 187–276
  2. J. A. Kaminski, V. M. Sloutsky, A. F. Heckler (2008), The advantage of abstract examples in learning math, Science 320: 454–455;重复研究见 D. De Bock 等 (2011),JRME 42(2);D. Trninic, M. Kapur, T. Sinha (2020),CogSci
  3. K. Kotovsky, J. R. Hayes, H. A. Simon (1985), Why are some problems hard? Evidence from Tower of Hanoi, Cognitive Psychology 17: 248–294
  4. E. R. Fyfe, N. M. McNeil, J. Y. Son, R. L. Goldstone (2014), Concreteness fading in mathematics and science instruction: A systematic review, Educational Psychology Review 26: 9–25
注:文中插图均为本文自制示意。图 6、图 9 的数字取自所列论文,图 7 的组合由本文程序穷举验证。
犹剑在心
北极甜虾(剑心犹在!)
— 全文完 —
Translated from the Chinese original, “学不会,是概念太难,还是讲法错了?”.
Same problem, new face: suddenly you get it

Can't Learn It?
Is It the Concept,
or the Teaching?

Bayes, four cards and the Lo Shu square
Some concepts, we like to say, are things “the human brain is just naturally bad at.” Yet write the same problem a different way, and the share of people who get it right can go from just over one in ten to nearly half. With two classic experiments from psychology, plus a little game you can play with a friend, this piece takes up one question: when you can't learn something, should you blame the concept, or the way it was explained?
Textbook versionPrevalence 1%Sick, positive 90%Healthy, positive 10%You test positive.Chance you're sick?Gut feeling: about 90%Picture 1,000 peopleTrue positive: 9False positive: 99Answer: 9 ÷ 108 ≈ 8.3%
Figure 1Two faces of the same problem. On the left, the textbook version; on the right, the same problem drawn as 1,000 people. The red dots are the 9 people who test positive and really are sick; the dark gray dots are the 99 who test positive but are actually healthy.

Let's start with a problem. Don't scroll down just yet.

A certain disease affects 1% of the population. There is a test for it: of the people who have the disease, 90% test positive; and even among people who don't, 10% get a false positive. Now your test comes back positive. What is the probability that you really have the disease?

Got your answer? Many people's first instinct is about 90%; after all, the test catches 90% of the people who are sick.

Now the same thing, put another way. Imagine 1,000 people taking the test. Of these, 10 are sick, and 9 of them test positive. The other 990 are healthy, and 99 of them get a false positive. So, of everyone who tests positive, what share is really sick?

This time it probably doesn't take long: 9 + 99 = 108 people test positive in all, and only 9 of them are really sick, about 8.3%.

The two passages say the same thing, and not a single number has changed. Yet the first version nudges people toward guessing 90%, while the second lets them see the answer almost at a glance.

This piece is about the question behind that phenomenon: when a student can't learn a concept, is the concept itself too hard, or did we pick the wrong way to teach it, and the wrong order?

Two explanations, two different predictions

“Bayesian reasoning goes against human intuition”: you have probably heard this before. What it implies is that the human brain was simply never designed to run on probability theory, so if you can't learn it, nobody is to blame.

The other explanation is that the human brain can do this kind of reasoning; we have just always been feeding it the problems in an awkward format.

The two may sound like a pointless squabble, but an experiment can actually tell which one is right. If the difficulty lies in the concept itself, writing the problem differently shouldn't change the results much; if it lies in the way the problem is written, a rewrite should make a clear difference. The two passages at the start of this piece were a small experiment of exactly this kind. Now for the big experiments that psychologists have run.

Bayes: switch to head counts

First, the bad news. According to a survey published in 1978 in the New England Journal of Medicine, researchers put a similar problem to students and doctors at Harvard Medical School, and only 11 of the 60 got it right. The most common wrong answer was 95%, dozens of times higher than the correct one. And according to a 1982 report by Eddy, when 100 doctors estimated the probability that a patient with a positive mammogram really had breast cancer, 95 of them gave figures of 70% to 80%; worked out with Bayes' formula, the answer is only about 7.5%.

Pile up enough results like these and you get the saying that “the human brain is not a Bayesian machine.”

In 1995, Gerd Gigerenzer and Ulrich Hoffrage did something very simple. They recruited 60 students at the University of Salzburg who had never learned Bayes' formula and gave them 15 problems of this kind. Each problem came in two versions: one written in probabilities (“the prevalence is 1%, the positive rate is 80%”), the other in head counts (“out of every 1,000 people, 10 have the disease, and 8 of them test positive”). They called the second kind natural frequencies.

The result: with the probability version, only 16% of the answers got there by Bayesian reasoning; with the head-count version, 46% did. And this was a strict measure: the answer had to be right, and the scratch work also had to show that the person really was reasoning the Bayesian way. Lucky guesses didn't count.

Later they tried the same thing with doctors. In a 1998 study, 48 doctors in Munich and Düsseldorf worked through four diagnostic problems. When the problems were written in probabilities, only 10% of their estimates were correct; in head counts, that rose to 46%.

1000people1%99%10sick990healthy9positive1negative99positive891negative90%10%
Figure 2The problem from the start, drawn as a tree. Start from 1,000 people, split them first into sick and healthy, then into positive and negative. The answer is the ratio of the head counts in the two highlighted boxes: 9 / (9 + 99).

Why do head counts make it easier? The two versions are exactly equivalent mathematically, but not computationally.

Gigerenzer and Hoffrage offered an analogy: the same number can be written in Arabic numerals or in Roman numerals, and the information is exactly the same. But try doing long division in Roman numerals. Feed Roman numerals to a calculator that only understands Arabic ones and it will spit out garbage; that doesn't mean it can't multiply.

The probability version makes you do three multiplications in your head, then an addition and a division, while keeping track along the way of which percentage refers to whom. The head-count version has done the multiplications for you in advance; all you have to do is count heads. In 1987 the cognitive scientists Larkin and Simon wrote a famous paper whose title says it all: “Why a Diagram is (Sometimes) Worth Ten Thousand Words.” It makes the same point: two representations can be informationally equivalent, yet one puts the answer in plain sight while the other makes you dig it out yourself.

What evidence is worth depends on both sides

Head counts also happen to correct a deeper error of intuition. When we weigh a piece of evidence, we tend to look at one side only: how many of the sick test positive? 90%. That's high, so a positive result must say a lot.

But what evidence is worth also depends on the other side: how many of the healthy test positive too? If healthy people often test positive as well, the evidence isn't worth as much.

Here's an everyday example. “This person often stays up late.” Does that suggest he's a programmer? Knowing that programmers like to stay up late isn't enough; you also have to ask whether people who aren't programmers stay up late. College students, doctors and people binge-watching shows all stay up late, so this is very weak evidence.

This is the habit that Bayesian reasoning is really meant to build: to weigh a piece of evidence, always compare how common it is under each of the two explanations. The formula just writes this habit down as an equation.

The math (optional)Bayes' formula has an easy-to-remember “odds” form. Let stand for “sick” and for “tests positive”:

The left side is the odds after seeing the evidence. On the right, the first factor is the prior odds, and the second is called the likelihood ratio, which is exactly the “weigh both sides” from above. The proof takes one line: divide by , and cancels.

Plug in the problem from the start: the prior odds are 10 : 990, and the likelihood ratio is 90% ÷ 10% = 9. Multiply them and you get 90 : 990, which is 1 : 11. Odds of 1 : 11 mean a probability of 1/12, about 8.3%, the same as counting heads.∎

Before: sick vs. healthy990 healthy10Positive: sick side × 90%, healthy side × 10%, so sick side × 9After a positive test990 healthy9090 : 990 = 1 : 11, probability 1/12 ≈ 8.3%
Figure 3The odds form. At first it is 10 sick people against 990 healthy ones; the evidence of a positive test multiplies the sick side by 9, giving 90 against 990, or 1 against 11. The evidence is strong, but the disease is just too rare.

Four cards: switch to social rules

The second experiment is even more famous. Time for another problem.

There are four cards on the table. Each has a letter on one side and a number on the other. The four sides you can see are A, K, 4 and 7. There is a rule: “If one side is a vowel, then the other side is an even number.” To check whether the rule has been broken, which cards, at minimum, must you turn over?

Most people turn over A and 4. The correct answer is A and 7.

Turning over A is right: if there's an odd number on its back, the rule is broken. Turning over 4 is pointless: the rule only says that a vowel must have an even number on its back, not that an even number must have a vowel on its back, so whatever letter is behind the 4, the rule holds. The card you really need to turn over is the 7: if there's a vowel on its back, the rule is broken.

This problem was devised by the British psychologist Peter Wason in 1966 and is known as the Wason selection task. According to a summary by Sperber and colleagues in 1995, in this standard abstract version usually only about one person in ten picks A and 7.

One more. You're an inspector at a bar, and the rule is: “If a person is drinking alcohol, then they must be at least 18.” In front of you are four people, and you know only one thing about each: one is drinking beer, one is drinking juice, one is 16 and one is 22. Who do you need to check?

This one takes almost no thought: check the beer drinker (to see whether they're of age) and check the 16-year-old (to see whether what they're drinking is alcohol).

If one side is a vowel, the other is evenAvowelKconsonant4even7oddIf drinking alcohol, must be 18 or overBeerdrinkingJuicenot drinkingAge 22adultAge 16minor
Figure 4The two problems match up one to one: A with the beer, K with the juice, 4 with the 22-year-old, 7 with the 16-year-old. In both, the ones to check are the two in red.

The two problems have exactly the same logical structure, yet their difficulty is worlds apart. According to an experiment by Griggs and Cox in 1982, once the problem was recast as a drinking-age rule, about 74% of college students in Florida got it right.

The math (optional)Let stand for “one side is a vowel” and for “the other side is even”; the rule is . The only counterexample to this rule is “ holds but doesn't”: a vowel on one side, an odd number on the other.

So what you're looking for is any card that might hide such a counterexample. For the card that shows (A), you need to see whether its back is ; for the card that shows (7), you need to see whether its back is . The other two cannot produce a counterexample, whatever is on their backs.

In other words, is equivalent to its contrapositive , but not to . People who pick 4 are in effect reading the rule as .∎

You may have noticed already that the mistakes in the two experiments look a lot alike.

BayesFour cardsAbstract formConditional probabilityIf … then …Common errorReading P(pos|sick)as P(sick|pos)Reading P → Qas Q → PNew faceCount headsCheck the barHabit to buildWeigh both sidesLook for violations
Figure 5Both mistakes get the direction backwards.

“Most sick people test positive” gets taken for “most people who test positive are sick”; “a vowel has an even number on its back” gets taken for “an even number has a vowel on its back.” Both mistakes read a relation that has a direction the wrong way round. Give the problem a new face, and the direction suddenly becomes clear.

It isn't “concreteness” that helps

Having read this far, it's easy to reach a simple conclusion: abstract things are hard, so switch to something concrete and everyday and all is well. The source material says so too. I thought the same at first, but after digging into the research, I think this conclusion is only half right.

Take Bayes first. Head counts do help, but not miraculously. A 2017 meta-analysis that pooled a large number of studies found that with probabilities, only 4% of people got the answer right on average; with head counts, the average rose only to 24%. In other words, even with head counts, three quarters of people still get it wrong. A 2018 study found an interesting reason: many people, once given head counts, turn them back into percentages to do the calculation, going round in a circle and ending up back on the old road.

Now the cards. A concrete story doesn't always help. In 1979, Manktelow and Evans ran five experiments, replacing the letters and numbers with all sorts of everyday content, and scores barely improved. Griggs and Cox themselves found that switching to a rule unfamiliar to college students (such as a certain postal regulation) didn't help either.

Conversely, a version that isn't concrete can help too. According to a 1985 report by Cheng and Holyoak, even with abstract rules, phrasing the rule as a “permission,” in the form “if you are to do a certain thing, you must first meet a certain condition,” led to 61% correct answers; an arbitrary rule about letters and numbers led to only 19%. The rule contained nothing concrete at all, yet scores still tripled.

0%20%40%60%80%Bayes: studentsGigerenzer & H. 1995Probability 16%Head counts 46%Bayes: doctorsHoffrage & G. 1998Probability 10%Head counts 46%Bayes: pooledMcDowell & J. 2017Probability 4%Head counts 24%Cards: permissionCheng & H. 1985Arbitrary rule 19%Permission 61%
Figure 6Accuracy before and after a change of format, in several studies. The studies differ in their problems, participants and scoring, so read each one only as a before-and-after comparison; don't compare levels across studies. The data come from the original papers or from accounts of them; see the references at the end.

Psychologists have argued for decades over why a change of format makes problems easier, and they still haven't settled it. Some say humans evolved a dedicated instinct for “cheater detection” (Cosmides 1989); some say we build up reasoning patterns for things like “permission” and “obligation” in everyday life (Cheng and Holyoak); still others say head counts work because they bring a relation of nested sets, “how many of those who test positive are really sick,” out into the open (Barbey and Sloman 2007).

These explanations differ in their details but point the same way: what helps is not “concreteness” as such. It's that the new format brings the structure of the problem into view, where it happens to match something your mind can already do. A concrete example helps if it does this; if it doesn't, it's useless, or even gets in the way.

“Gets in the way” is no exaggeration. A 2008 study in the journal Science had students learn a mathematical concept through concrete examples such as measuring cups and pizzas, while another group learned it with abstract symbols. When they were then tested in a new setting, the abstract group did better. The concrete details were too eye-catching: the students remembered the measuring cups, not the structure. I should be clear, though: two later attempts to replicate the study failed to get the same result, and it is still contested.

One game, three faces

At this point, here's an example you can try for yourself.

A little game: Make 15 (play first, then read on)
  1. Find a friend and lay out the nine numbers from 1 to 9 on the table.
  2. Take turns picking up one number at a time; whatever you pick up is yours.
  3. The first player to hold three numbers that add up to exactly 15 wins. If all nine numbers are gone and nobody has made 15, it's a draw.

Play a couple of rounds and you'll find this game takes real brainpower. Every time you take a number, you have to work out how far each pair in your hand is from 15, and which set your opponent is about to complete.

Now for the reveal: this game is really just tic-tac-toe.

Tic-tac-toeLo ShuMake 15492357816You took258Friend took472 + 5 + 8 = 15, all in a lineOnly 8 sets of three add up to 15:1 5 91 6 82 4 92 5 82 6 73 4 83 5 74 5 6
Figure 7Three faces of the same game. Arrange 1 to 9 in a 3×3 grid the way the Lo Shu does, and every row, every column and both diagonals add up to 15; what's more, every set of three numbers that adds up to 15 lies on a line. Playing Make 15 is playing tic-tac-toe on this board.

Arrange 1 to 9 in the grid above and you have what the ancient Chinese called the Lo Shu square, which is also the smallest magic square. It has this property: three different numbers add up to 15 if and only if they form a line in the grid. So “taking a number” is “placing a piece,” and “making 15” is “getting three in a row.”

The math (optional)Choose three different numbers from 1 to 9. Only 8 such sets add up to 15:

{1,5,9}, {1,6,8}, {2,4,9}, {2,5,8}, {2,6,7}, {3,4,8}, {3,5,7}, {4,5,6}.

The 3 rows, 3 columns and 2 diagonals of the Lo Shu also make exactly 8 lines, and if you check them one by one, they are precisely these 8 sets, not one more and not one less. You can also read off which numbers are worth the most: 5 appears in 4 of the sets, matching the center of the board; 2, 4, 6 and 8 each appear 3 times, matching the four corners; 1, 3, 7 and 9 each appear twice, matching the four edges. In tic-tac-toe you take the center first; in Make 15 you should take the 5 first.∎

Draw the game as a grid, and even small children can play it, and they all know to take the middle first; write it as numbers, and even adults have to count on their fingers. The rules, who wins and who loses, the best strategy: none of it has changed. The only thing that has changed is how it looks.

Psychologists experimented with such “isomorphs” long ago. The best known involves the Tower of Hanoi: in 1985, Kotovsky, Hayes and Simon rewrote the Tower of Hanoi as stories like “alien monsters swapping globes.” The structure was exactly the same, yet some versions were much harder.

Order: grow the intuition first, then give it a name

Besides representation, there is also the question of order.

The way of teaching we know best goes like this: first the definition, then the theorem, then worked examples, and finally applications. With Bayes, that means writing out the formula first, then plugging the medical-test problem into it.

The other road runs the opposite way. First you get a real problem and guess by intuition, and you guess wrong; then the problem is written a different way, and you spot the pattern yourself; once the pattern has taken hold, you are told its name and how to write it as a formula; finally you are taken to a few completely different settings and recognize it there.

The usual orderRuleTheoremExamplesApplicationsThe other roadPuzzleWrongRecastPatternRuleExtendWhere the rule comes in: first, or after you understand
Figure 8Two orders of teaching. In the top one, the rule comes first; in the bottom one, by the time the rule appears, you already know what it is saying.

On the second road, a formula is no longer a string of symbols to memorize but a description of something you already understand. I've written before about the research on “problem first, explanation after”, so I won't repeat it here. One thing worth adding is an idea from education research called concreteness fading: start with something concrete, replace it step by step with more abstract representations, and finally take the concrete crutch away. As we saw earlier, concrete examples sometimes leave people remembering only the surface; that last step, “taking the crutch away,” is there to guard against exactly this.

Three yardsticks

To call a way of teaching “good,” you first have to say what you are measuring it with. There are at least three yardsticks:

The first is immediate learning: right after the lesson, can you get it right? The second is retention: a week or a month later, can you still do it? The third is transfer: when the problem moves from medical tests to courtroom evidence or scientific experiments, can you still recognize it as the same thing?

These three yardsticks often give different readings.

In 2001, Peter Sedlmeier and Gigerenzer wrote a computer tutorial with a very plain title: “Teaching Bayesian Reasoning in Less Than Two Hours.” They compared two ways of teaching: one taught students to plug probabilities into Bayes' formula (“applying the formula”), the other taught them to redraw the problems as trees or grids of head counts (“changing the format”). The students were tested right after the lesson, again a week later, and again five weeks later.

0%25%50%75%100%BeforeRight after1 week5 weeksFormula: 20%30%60%Freq. tree 90%Freq. grid 75%100%Groups of a dozen or more, 10 problems, median correct
Figure 9University of Chicago students, tested on 10 problems each time; the vertical axis is the median number of problems answered correctly (as a percentage). The formula group fell back to 20% after five weeks; the two groups that drew head counts didn't drop. Each group started with 14 to 20 people, and some were absent from the last two tests. Data read from Figure 9 of Sedlmeier and Gigerenzer's 2001 paper (lenient scoring).

On the immediate yardstick, both ways of teaching worked very well: the formula group rose from 0 to 60%, and the two head-count groups rose to 75% and 90%.

On the retention yardstick, the difference shows. Five weeks later the formula group had fallen back to 20%, almost as if they had never been taught; the head-count groups hadn't slipped at all, and one of them was even higher.

I should be honest here: quite a few people missed the later tests in this Chicago round. By week five, the frequency-tree group was down to 5 people, so this round on its own isn't solid enough. That's why they ran it again in Berlin, with 56 people and not a single dropout: the formula group scored 64% right after the lesson and 43% five weeks later; the frequency-tree group scored 86% right after the lesson and still 86% five weeks later.

On the transfer yardstick, the result surprised me: both ways of teaching transferred well, and on new problems the students had never seen, their scores were about the same as on the ones they had practiced. The source material says that “students who learn to apply the formula may not have learned a way of thinking that transfers,” but this study doesn't support it.

Still, the “transfer” here is near transfer: the new problems were still standard Bayesian word problems. As for far transfer, from medical tests to courtroom evidence or everyday judgments, I couldn't find a study that has seriously tested it. That may well be the experiment most worth doing.

Can intuition be built?

The source material ends with a question: can mathematical intuition be built systematically, rather than stumbled upon by luck after piles of practice?

My answer: part of it can, and the strongest evidence is about whether it sticks.

Writing a concept a different way can take it from “only a few people get it” to “lots of people get it,” and they still remember it five weeks later. That shows that much of what we take for “just not wired for it” is really “not shown the right format.” On this point the source material is right: Bayesian reasoning is not all that counterintuitive; we have simply been teaching it in a counterintuitive way.

But we shouldn't overstate it either. Even the best format only gets about half of people to the right answer, and the meta-analysis average is lower still; concrete examples sometimes help and sometimes get in the way; and for the step of “recognizing that two problems are really the same problem,” no one has yet found a reliable way to teach it. The concept's own difficulty is real; it's just smaller than we thought.

If I had to give one piece of advice to people who teach math, it would be this: don't teach just one face. Show students at least two or three different faces of the same structure, and let them discover for themselves that the same thing lies underneath. Medical tests and late nights, vowels and beer, tic-tac-toe and Make 15. When, one day, somewhere they have never been before, they recognize that familiar structure on their own, that's when they have truly learned it.

In closing

Next time you, or your child, just can't learn some concept no matter how hard you try, don't rush to conclude, “I'm not cut out for this.”

Maybe what you're missing isn't brains, but another face.

Often,
the hard part isn't the concept but how it looks.
Let's talk in the comments

Have you ever had this happen: you struggled with a concept for ages, then someone explained it differently and it suddenly clicked? Which concept was it, and what was the new explanation?

References
  1. G. Gigerenzer, U. Hoffrage (1995), How to improve Bayesian reasoning without instruction: Frequency formats, Psychological Review 102(4): 684–704
  2. U. Hoffrage, G. Gigerenzer (1998), Using natural frequencies to improve diagnostic inferences, Academic Medicine 73(5): 538–540
  3. P. Sedlmeier, G. Gigerenzer (2001), Teaching Bayesian reasoning in less than two hours, Journal of Experimental Psychology: General 130(3): 380–400
  4. W. Casscells, A. Schoenberger, T. B. Graboys (1978), Interpretation by physicians of clinical laboratory results, New England Journal of Medicine 299: 999–1001
  1. D. M. Eddy (1982), Probabilistic reasoning in clinical medicine: Problems and opportunities, in Judgment under Uncertainty: Heuristics and Biases (Cambridge University Press)
  2. M. McDowell, P. Jacobs (2017), Meta-analysis of the effect of natural frequencies on Bayesian reasoning, Psychological Bulletin 143
  3. P. Weber, K. Binder, S. Krauss (2018), Why can only 24% solve Bayesian reasoning problems in natural frequencies, Frontiers in Psychology 9: 1833
  1. A. K. Barbey, S. A. Sloman (2007), Base-rate respect: From ecological rationality to dual processes, Behavioral and Brain Sciences 30(3)
  2. J. H. Larkin, H. A. Simon (1987), Why a diagram is (sometimes) worth ten thousand words, Cognitive Science 11: 65–99
  3. P. C. Wason (1968), Reasoning about a rule, Quarterly Journal of Experimental Psychology 20(3): 273–281
  4. D. Sperber, F. Cara, V. Girotto (1995), Relevance theory explains the selection task, Cognition 57: 31–95
  1. R. A. Griggs, J. R. Cox (1982), The elusive thematic-materials effect in Wason's selection task, British Journal of Psychology 73: 407–420
  2. K. I. Manktelow, J. St. B. T. Evans (1979), Facilitation of reasoning by realism: Effect or non-effect?, British Journal of Psychology 70: 477–488
  3. P. W. Cheng, K. J. Holyoak (1985), Pragmatic reasoning schemas, Cognitive Psychology 17(4): 391–416
  1. L. Cosmides (1989), The logic of social exchange, Cognition 31: 187–276
  2. J. A. Kaminski, V. M. Sloutsky, A. F. Heckler (2008), The advantage of abstract examples in learning math, Science 320: 454–455; replications: D. De Bock et al. (2011), JRME 42(2); D. Trninic, M. Kapur, T. Sinha (2020), CogSci
  3. K. Kotovsky, J. R. Hayes, H. A. Simon (1985), Why are some problems hard? Evidence from Tower of Hanoi, Cognitive Psychology 17: 248–294
  4. E. R. Fyfe, N. M. McNeil, J. Y. Son, R. L. Goldstone (2014), Concreteness fading in mathematics and science instruction: A systematic review, Educational Psychology Review 26: 9–25
Note: All figures are schematics made by the author. The numbers in Figures 6 and 9 are taken from the papers listed, and the combinations in Figure 7 were verified by exhaustive search with a program written for this piece.
犹剑在心
北极甜虾(剑心犹在!)
— The End —

← All essays