Essays北极甜虾 · No. 4

字
DATA STORY数据 · 长文
把《全唐诗》一个字一个字地数一遍

四万八千首唐诗里,
诗人最常写的是什么?

用数据读一遍《全唐诗》
康熙年间编成的《全唐诗》,收诗四万八千九百余首。我把它整部交给计算机,一个字一个字地数了一遍:出现最多的字是哪个?最常见的两个字是什么?李白和杜甫的用字差在哪里?白居易真的“老妪能解”吗?数完才发现,诗人写得最多的,是他们没有的东西。
不26,007次 · 全唐诗里出现最多的字平均每 100 个字里就有 1 个人山无风一日云有来天何第 2–12 名
全唐诗里出现最多的字。下面一行是第 2 到第 12 名,条的高度按出现次数画。

先请你猜一猜:在全部唐诗里,出现次数最多的字是哪一个?

月?酒?愁?还是“床前明月光”里的某个字?

都不是。答案是“不”。

这篇文章用的是开源的《全唐诗》数据,统计范围是康熙年间编定的九百卷:去掉重复后 49,037 首,256 万字。怎么数的、哪些不算,文末有一段“统计口径”。

01
THE CORPUS

先数一数这部书

《全唐诗》是康熙四十四年(1705 年)下令编的。彭定求等十位翰林奉旨编校,曹雪芹的祖父曹寅在扬州主持刊刻,不到两年就编成了九百卷,自称“得诗四万八千九百余首,凡二千二百余人”。

49,037首诗去掉 905 首重出诗之后
256 万个汉字共 458,639 句
8,653个不同的字1,506 个只出现一次
2,617个署名另有不少佚名之作
这一次统计的规模(康熙《全唐诗》九百卷)。

里面最常见的是哪种诗?按句数和每句的字数粗分一下:五言八句最多,占 28.8%;七言四句次之,占 20.3%。像《静夜思》那样的五言四句,只占 6.8%。

五言八句28.8%七言四句20.3%五言其他17.7%七言八句15.9%杂言7.6%五言四句6.8% 《静夜思》这一类七言其他2.9%
按句数和字数粗分的诗体比例。只看形状、不查平仄,所以“五言八句”并不都是严格的五律。
02
THE TOP WORD

排第一的,是“不”

“不”字一共出现了 26,007 次,平均每 100 个字里就有 1 个;每 10 首唐诗里,大约有 4 首用到了它。

10,00020,000不26,007人20,659山15,892无15,599风15,392一15,017日14,731云12,560有12,514来12,119天12,025何12,020中11,242时11,178花11,140上10,940春10,908水10,908月10,791相9,534
全唐诗里出现次数最多的 20 个字(按繁体原字统计,以简体显示;“云”指云彩的“雲”)。

排在前面的,还有一批和它同类的字:“无”排第 4,“何”排第 12。把“不、无、未、莫、非”这几个表示否定的字算在一起,62.6% 的唐诗里至少有一个。

把两个字连起来看,就更明显了。全唐诗里最常见的两字组合,第一名是“何处”,出现在 1,498 首诗里。前十名里,“不知”“不见”“不可”“不得”占了四个,“万里”“千里”又占了两个。

含“不”“无”“何”的组合其他5001,0001,500何处1,515万里1,430不知1,422千里1,289今日1,146不见1,135不可1,126白云1,108春风1,103不得937人间872明月871无人869风吹817惆怅775故人770秋风735悠悠731相思723长安709
最常见的 20 个两字组合(只算同一句里相邻的两个字)。“何处”第一,政治大学刘昭麟等人 2015 年的研究也得出了同样的排名。

把这些放在一起读:不知道,看不见,不可以,得不到;问在哪里;隔着万里千里。诗人最常写的,不是眼前有什么,而是想要却没有的:人不在身边,家回不去,时间留不住。

情绪字也是这样。“愁”出现了 4,245 次,是“喜”(1,288 次)的 3.3 倍。

诗,是从“得不到”里长出来的。

03
WHAT THEY SAW

春天、白色和千万

去掉“不、之、而、何”这些虚字,剩下的实字里排在前面的是:人、山、风、日、云、来、天、中、时、花、上、春、水、月。“月”排在全部字的第 19 位,每 5 首唐诗里就有 1 首写到月亮。

我又挑了三组字来看:季节、颜色和数字。

四季
5,00010,000春10,908夏921秋7,945冬513
颜色
2,0004,0006,0008,000白8,368青6,408红4,049黄3,738碧2,747绿2,728翠2,499紫2,146丹1,873朱1,406赤744黑506
数字
5,00010,00015,000一15,017二三5,971四五六七八九十百千万5,955 / 5,612
三组字的出现次数。颜色和数字都按字计,所以“黄”里算进了“黄河”“黄昏”,“一”里算进了“一片”“一声”。

春天压倒性地多。“春”出现了 10,908 次,是“夏”的近 12 倍、“冬”的 21 倍;“秋”排第二,7,945 次。诗人偏爱的,是春和秋这两个变化的季节:一个开始,一个凋零。

颜色里,“白”遥遥领先,8,368 次:白云、白日、白发。最少的是“黑”,只有 506 次。

数字最有意思。除了“一”,用得最多的是“三”“千”“万”,三个字几乎一样多,都在 5,600 到 6,000 次之间,远远多过“二”“四”“七”“八”。诗人不爱精确,爱的是“三”这样的虚数,和“千”“万”这样的夸张:万里,千里。“二”只有 2,002 次,不过“两”和“双”分担了很多“二”的活儿,各有两千多次。

04
VOCABULARY

认多少字,能读唐诗

全唐诗一共用到了 8,653 个不同的字,但字和字的地位很不平等。最常用的 253 个字,就占了全部用字的一半;认识最常用的 917 个字,唐诗里八成的字你都认得;1,564 个字覆盖九成。另一头,有 1,506 个字在整部全唐诗里只出现过一次。

25%50%75%100%1101001,0001 万按出现次数排名(取前 N 个字)前 253 个字:一半前 917 个字:八成前 1,564 个字:九成前 3,992 个字:99%
按出现次数从高到低排列,前 N 个字合起来占全部用字的比例。横轴是对数刻度。

这种“少数字包办大部分”的规律,语言学里叫齐夫定律:一个字的排名越靠后,出现的次数就越少,而且按幂律递减。

05
LI BAI × DU FU

李白和杜甫,两本字典

比较两位诗人,不能只比次数。杜甫名下的字比李白多,约 10.6 万 字对 8.4 万 字,什么字都会多一些。我用了统计里常用来比较两份文本的一种方法,叫带先验的对数几率比(Monroe 等人,2008)。它会扣掉总字数的差别,也会照顾到每个字本来有多常见。结果是这样的:

李白偏爱杜甫偏爱月老君病海江去近金衰绿峡笑野言雨之旧玉细条越长,偏好越明显
和对方相比,各自最偏爱的 10 个字。条越长,偏好越明显(z 值);换不同的先验强度,前十名基本不变。

李白的字典里,是月、海、天、金、玉、笑:37% 的李白诗写到了月亮,杜甫只有 15%。杜甫的字典里,是老、病、衰、江、峡、雨:近四分之一的杜甫诗里有“老”字;“病”字杜甫用了 162 次,李白只有 8 次。一个望着天上的月亮和大海,一个守着眼前的江水和病体。

还有一个意外:论写酒,白居易比李白更勤。20.6% 的白居易诗里有“酒”字,李白是 16.7%,全唐诗平均只有 9.4%。

06
THE MISSING WORDS

数据里的避讳

上面那张图里,李白偏爱的字里有一个“言”。这个字李白用了 135 次,杜甫只用了 3 次。

这就怪了。按全唐诗的平均用字率,杜甫名下那十万多字里,“言”应该出现 145 次左右,实际只有 3 次。

杜甫的祖父,名叫杜审言。

按平均用字率应有的次数杜甫诗里实际的次数050100150200言应有 145,实有 3闲应有 194,实有 2海棠应有 1.7,实有 0
杜甫诗里几个字(词)的实际次数,和按全唐诗平均用字率算出的应有次数。

古人有避讳的规矩:说话写字,要避开长辈的名字。杜甫避父亲的讳是有名的。他父亲叫杜闲,前人早就注意到,杜诗里几乎找不到“闲”字,个别例外也被考证为后来传刻的错字。数据也是这样:按平均应有约 190 次,实际只有 2 次。

“言”字不一样。唐律有“二名不偏讳”的规定:长辈的名字是两个字时,单用其中一个字并不算犯讳。也就是说,杜甫本来不必避“言”字。但在他的诗里,“言”几乎消失了。

唐人对家讳,常常比法律要求的更严。李贺的父亲名叫晋肃,就有人说李贺不该去考“进士”,韩愈为此专门写了一篇《讳辩》,引的正是“二名不偏讳”这一条。

杜甫是不是在避祖父的讳?数据只能给出线索,给不出定论。这一点我没有在常见的资料里查到专门的讨论,欢迎懂行的朋友指正。

作为对照,还有一个流传很广的说法:杜甫的母亲名叫海棠,所以他从不写海棠。数据查不出什么:整部全唐诗里“海棠”一共只出现了 41 次,按杜甫的字数,他本来也只该写到一两次。没写,并不反常。

说一个字“少”,要先问它本该有多少。

07
BAI JUYI

白居易真的老妪能解吗

宋人笔记里有个故事,说白居易每写一首诗,先念给一位老太太听:

白乐天每作诗,令一老妪解之,问曰:“解否?”妪曰解,则录之;不解,则易之。—— 惠洪《冷斋夜话》

这个故事未必是真的,前人就有怀疑。不过如果它说得对,白居易用的字应该最“平常”。我拿全唐诗里最常用的 1,000 个字当作“常用字”,算出每位诗人有多少比例的字落在常用字之外,比较的是名下至少有 200 首诗的 61 位诗人。

10%15%20%25%30%61 位诗人的中位数 17.0%赵嘏11.0%刘长卿12.5%张籍14.1%李白16.2%王维16.7%孟浩然17.4%白居易17.9%刘禹锡18.5%杜牧18.8%李商隐20.1%杜甫21.8%李贺22.1%元稹23.7%陆龟蒙24.0%皮日休25.4%韩愈28.2%
常用 1,000 字以外的字所占的比例(选取 16 位诗人;横线是按诗重抽样得到的 95% 区间)。

结果有点意外:白居易是 17.9%,在 61 位诗人里排第 36,比中位数(17.0%)还“冷”一点,也比李白(16.2%)冷。最“平常”的是赵嘏(11.0%),最“冷”的是韩愈(28.2%),皮日休、陆龟蒙、元稹、李贺、杜甫紧随其后。

但只看比例会看错。把每个人在常用字之外用得最多的字拿出来,差别就清楚了:

白居易冷字比例 17.9%
劝慵勿睡富彼妻味饱司退酌
多是过日子的字
韩愈冷字比例 28.2%
汝余视谓固罪杂腾施颠母鬼
多有古文的口气
李贺冷字比例 22.1%
母蒲铜团蛾踏尾刺碎脉娥虫
幽冷的物象
杜甫冷字比例 21.8%
汝戈巫甲惨坤冬适焉黑涛奔
兵戈与巫峡
四位诗人在常用 1,000 字以外用得最多的 12 个字,按次数排列,没有挑选。

白居易的“冷字”是睡、饱、味、妻、慵、酌:吃饭睡觉,喝酒犯懒,全是过日子的字。这些字在诗里少见,在生活里天天都用,老太太当然听得懂。韩愈的“冷字”是汝、余、谓、固,这些是写文章的字,正好应了后人说他“以文为诗”。李贺是铜、蛾、碎、虫,阴冷幽怪。杜甫是戈、甲、巫、涛,兵荒马乱,又困在巫峡。

所以,“老妪能解”量的是生活里的常用字,不是诗里的常用字。用诗的尺子量,白居易并不最浅。

白居易的“冷”,冷在把日子写进了诗。

08
MY TAKE

数字会数,不会读

数完这四万多首诗,说说我的几点看法。

一、数据最擅长的,是看见“没有”。

人读诗,读的是写下来的字;没写的字,人很难注意到。杜甫诗里没有“言”,读一千遍也未必发现,计算机数一遍就看出来了。

二、但数字不会读诗。

“不”是唐诗里最多的字,可单拿出来,它一点诗意也没有。“不知”“不见”之所以动人,靠的是后面跟着的东西:“不知何处是他乡”,“不见长安见尘雾”。

三、尺子决定结论。

用“诗里的常用字”量,白居易不算浅;换成“生活里的常用字”,结论可能就反过来了。任何统计,都是先选好了一把尺子。看结论之前,先看尺子。

我之前写过理性和感性。这一期是“理”的那一半:数据能告诉我们诗人写了什么,写了多少;至于为什么动人,还得靠读的人自己去感受。

09
CODA

写在最后

四万多首诗,二百五十多万字,数到最后,排在第一的是一个“不”,最常问的是一句“何处”。

“诗人最常写的,
是他们没有的东西。
✎ 留言区聊聊

你最喜欢哪一位唐代诗人?在留言区告诉我,下次帮你数一数他(她)最偏爱哪些字。

附表:全唐诗最常用的 50 个字
不26,007
人20,659
山15,892
无15,599
风15,392
一15,017
日14,731
云12,560
有12,514
来12,119
天12,025
何12,020
中11,242
时11,178
花11,140
上10,940
春10,908
水10,908
月10,791
相9,534
年9,404
为9,379
生9,295
君9,190
长9,018
心8,936
自8,909
如8,631
知8,545
白8,368
归8,217
此8,011
秋7,945
行7,828
见7,736
去7,671
清7,455
夜7,370
在7,303
江7,291
空7,286
得7,225
下7,199
高7,123
未6,892
明6,683
多6,533
金6,521
门6,484
青6,408
统计口径
  • 数据:开源项目 chinese-poetry 的《全唐诗》部分(GitHub,MIT 许可,2023 年 10 月版)。
  • 范围:只取数据集的前 49,942 条,对应康熙《全唐诗》九百卷(含卷末的词)。数据集后面附的《全唐诗逸》(日本人市河世宁编)和今人编的《全唐诗补编》等补录材料,不计入。
  • 清洗:去掉括号里的校注和异文;缺字、拼合字不计;写法不同、意思相同的异体字合并(如“回”和“迴”、“峰”和“峯”),意思不同的分开统计(如云彩的“雲”和“说”的“云”)。
  • 去重:正文完全相同的 905 首重出诗只算一次,剩 49,037 首。比较诗人时,仍按全唐诗的署名计入各自名下。
  • 单位:字频按出现次数;“百分之几的诗”按至少出现一次计;组诗的每一首单独计(若把组诗合并成一首,约 42,400 首)。统计用繁体原字,展示时转成简体。
  • 方法:诗人的用字偏好用带先验的对数几率比(Monroe 等,2008),先验强度换成 2,000 或 50,000,前十名基本不变。“常用字”取全唐诗里最常用的 1,000 个字;比例的 95% 区间按诗重抽样 1,000 次得到。
  • 局限:字不等于词;颜色、数字按字计;全唐诗本身有误收、重出和归属存疑的诗;网上和学界已有不少类似的字频统计,总体结论相近。
参考资料
  1. chinese-poetry:中华古典诗词数据库(GitHub,MIT 许可),《全唐诗》部分
  2. 彭定求等编《全唐诗》(中华书局 1960 年排印本,附市河世宁《全唐诗逸》);陈尚君辑校《全唐诗补编》(中华书局,1992)
  3. B. L. Monroe, M. P. Colaresi & K. M. Quinn, Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict, Political Analysis 16(4), 2008
  4. Chao-Lin Liu et al., Color Aesthetics and Social Networks in Complete Tang Poems, PACLIC 29, 2015
  5. 韩愈《讳辩》;惠洪《冷斋夜话》卷一;《礼记·曲礼上》
  6. 李白《客中行》;白居易《长恨歌》
注:文中图表均为本文根据上述数据自行统计、绘制。
犹剑在心
北极甜虾(剑心犹在!)
— 全文完 —
Translated from the Chinese original, “四万八千首唐诗里,诗人最常写的是什么?”.
字
DATA STORYData · Long read
Counting the “Complete Tang Poems,” one character at a time

In 48,000 Tang Poems,
What Did Poets Write About Most?

Reading the “Complete Tang Poems” through data
The “Complete Tang Poems,” compiled under the Kangxi Emperor, holds more than 48,900 poems. I handed the whole thing to a computer and counted it, character by character. Which character appears most often? What is the most common pair of characters? Where do Li Bai and Du Fu differ in the characters they use? Is it true that “even an old woman could understand” Bai Juyi? Only after the counting was done did I see it: what poets wrote about most was what they did not have.
不26,007times · top character1 in every 100 characters人山无风一日云有来天何Ranks 2–12
The characters that appear most often in the “Complete Tang Poems.” The row below shows ranks 2 to 12, with bar heights scaled to the number of occurrences.

First, take a guess: across all Tang poems, which character appears most often?

月 (moon)? 酒 (wine)? 愁 (sorrow)? Or one of the characters in “床前明月光” (“Before my bed, the bright moonlight”)?

None of these. The answer is “不” (not).

This article uses open-source data for the “Complete Tang Poems” and covers the 900 volumes compiled under Kangxi: 49,037 poems once duplicates are removed, 2.56 million characters. How I counted and what I left out are set out in the “How I counted” section at the end.

01
THE CORPUS

First, Sizing Up the Book

The “Complete Tang Poems” was commissioned in the 44th year of the Kangxi reign (1705). Peng Dingqiu and nine other Hanlin academicians edited it by imperial order, and Cao Yin, grandfather of Cao Xueqin, oversaw the printing in Yangzhou. In under two years all 900 volumes were done, gathering, by its own count, “more than 48,900 poems by more than 2,200 poets.”

49,037poemsafter removing 905 duplicates
2.56 millioncharactersin 458,639 lines
8,653distinct characters1,506 appear only once
2,617named poetsplus many anonymous poems
The scale of this count (the Kangxi “Complete Tang Poems,” 900 volumes).

What kind of poem is most common? A rough sort by number of lines and characters per line: eight lines of five characters is the most common, at 28.8%, followed by four lines of seven characters, at 20.3%. Four lines of five characters, the form of “Quiet Night Thoughts,” make up only 6.8%.

5 × 828.8%7 × 420.3%5 × other17.7%7 × 815.9%Mixed7.6%5 × 46.8%, like “Quiet Night Thoughts”7 × other2.9%
Poetic forms, sorted roughly by characters per line and number of lines (5 × 8 means eight lines of five characters). Only the shape is counted, not the tonal pattern, so not every 5 × 8 poem is a strict five-character regulated verse.
02
THE TOP WORD

In First Place: 不 (Not)

“不” (not) appears 26,007 times in all, about once in every 100 characters, and roughly 4 in every 10 Tang poems use it.

10,00020,000不26,007人20,659山15,892无15,599风15,392一15,017日14,731云12,560有12,514来12,119天12,025何12,020中11,242时11,178花11,140上10,940春10,908水10,908月10,791相9,534
The 20 most frequent characters in the “Complete Tang Poems” (counted in the original traditional characters and shown in simplified form; “云” here is “雲,” cloud).

Higher up the list are more characters of the same kind: “无” (without) ranks 4th and “何” (what, where) 12th. Taken together, the negating characters 不 (not), 无 (without), 未 (not yet), 莫 (do not) and 非 (is not) appear at least once in 62.6% of Tang poems.

Look at pairs of characters and it becomes even clearer. The most common two-character combination in the “Complete Tang Poems” is “何处” (where), found in 1,498 poems. In the top ten, “不知” (don’t know), “不见” (can’t see), “不可” (cannot) and “不得” (can’t get) take four places, and “万里” (ten thousand miles) and “千里” (a thousand miles) take two more.

Pairs with 不, 无 or 何Others5001,0001,500何处1,515万里1,430不知1,422千里1,289今日1,146不见1,135不可1,126白云1,108春风1,103不得937人间872明月871无人869风吹817惆怅775故人770秋风735悠悠731相思723长安709
The 20 most common two-character combinations (counting only adjacent characters within a line). “何处” comes first, the same ranking found in a 2015 study by Chao-Lin Liu and colleagues at National Chengchi University.

Read them together: don’t know, can’t see, cannot, can’t get; asking where; ten thousand miles, a thousand miles away. What poets write about most is not what is in front of them, but what they want and do not have: someone who isn’t there, a home they can’t go back to, time that won’t stay.

Words for feelings tell the same story. “愁” (sorrow) appears 4,245 times, 3.3 times as often as “喜” (joy, 1,288 times).

Poetry grows out of what we cannot have.

03
WHAT THEY SAW

Spring, White and Thousands

Set aside function characters such as 不, 之, 而 and 何, and the content characters at the top are 人 (person), 山 (mountain), 风 (wind), 日 (sun), 云 (cloud), 来 (come), 天 (sky), 中 (in), 时 (time), 花 (flower), 上 (on), 春 (spring), 水 (water) and 月 (moon). “月” ranks 19th among all characters: one Tang poem in five mentions the moon.

I also picked three groups of characters to look at: seasons, colors and numbers.

Seasons
5,00010,000春10,908夏921秋7,945冬513
Colors
2,0004,0006,0008,000白8,368青6,408红4,049黄3,738碧2,747绿2,728翠2,499紫2,146丹1,873朱1,406赤744黑506
Numbers
5,00010,00015,000一15,017二三5,971四五六七八九十百千万5,955 / 5,612
Occurrences in the three groups. Colors and numbers are counted by character, so “黄” (yellow) includes “黄河” (the Yellow River) and “黄昏” (dusk), and “一” (one) includes “一片” (a stretch of) and “一声” (a single sound).

Spring wins by a landslide. “春” (spring) appears 10,908 times, nearly 12 times as often as “夏” (summer) and 21 times as often as “冬” (winter); “秋” (autumn) comes second, with 7,945. What poets love are the two seasons of change, spring and autumn: one a season of beginnings, the other of withering.

Among colors, “白” (white) is far ahead, at 8,368: white clouds, the white sun, white hair. The least used is “黑” (black), with only 506.

Numbers are the most interesting. Apart from “一” (one), the most used are “三” (three), “千” (thousand) and “万” (ten thousand), almost neck and neck at between 5,600 and 6,000 each, and far ahead of “二” (two), “四” (four), “七” (seven) and “八” (eight). Poets have no love for precision. What they love is a loose “three,” meaning “several,” and the exaggeration of “thousand” and “ten thousand”: ten thousand miles, a thousand miles. “二” appears only 2,002 times, though “两” and “双” (both meaning “two” or “a pair”) take on much of its work, with more than two thousand occurrences each.

04
VOCABULARY

How Many Characters Does It Take to Read Tang Poetry?

The “Complete Tang Poems” uses 8,653 distinct characters in all, but characters are far from equal. The 253 most common account for half of all the text; know the 917 most common, and you will recognize eight in ten of the characters you meet in Tang poetry; 1,564 characters cover nine in ten. At the other end, 1,506 characters appear exactly once in the whole collection.

25%50%75%100%1101001,00010,000Rank by frequency (top N characters)Top 253: halfTop 917: 80%Top 1,564: 90%Top 3,992: 99%
The share of all character occurrences covered by the top N characters, ranked from most to least frequent. The horizontal axis is on a log scale.

This pattern, in which a few characters do most of the work, is known in linguistics as Zipf’s law: the further down the ranking a character sits, the less often it appears, and the counts fall off as a power law.

05
LI BAI × DU FU

Li Bai and Du Fu: Two Dictionaries

To compare two poets, it isn’t enough to compare counts. Du Fu left more text than Li Bai, about 106,000 characters against 84,000, so nearly every character shows up more often in his poems. I used a method that statisticians often use to compare two texts, the log-odds ratio with an informative prior (Monroe et al., 2008). It corrects for the difference in total length and also allows for how common each character is to begin with. Here is the result:

Li Bai favorsDu Fu favors月老君病海江去近金衰绿峡笑野言雨之旧玉细Longer bar = stronger preference
The 10 characters each poet favors most compared with the other. The longer the bar, the stronger the preference (z-score); with different prior strengths, the top ten barely change.

Li Bai’s dictionary holds 月 (moon), 海 (sea), 天 (sky), 金 (gold), 玉 (jade) and 笑 (laugh): 37% of Li Bai’s poems mention the moon, against only 15% of Du Fu’s. Du Fu’s dictionary holds 老 (old), 病 (ill), 衰 (decline), 江 (river), 峡 (gorge) and 雨 (rain): nearly a quarter of Du Fu’s poems contain “老,” and Du Fu used “病” 162 times, Li Bai only 8. One gazes at the moon in the sky and at the sea; the other keeps watch over the river before his eyes and his own failing body.

One more surprise: when it comes to wine, Bai Juyi wrote about it more often than Li Bai. “酒” (wine) appears in 20.6% of Bai Juyi’s poems, against 16.7% of Li Bai’s and just 9.4% across the “Complete Tang Poems” as a whole.

06
THE MISSING WORDS

A Taboo in the Data

Among the characters Li Bai favors in that chart is “言” (speak). Li Bai used it 135 times; Du Fu used it only 3 times.

That is odd. At the average rate for the “Complete Tang Poems,” the hundred thousand or so characters under Du Fu’s name should include “言” about 145 times. The actual count is 3.

Du Fu’s grandfather was named Du Shenyan (杜审言).

Expected at average rateActual count in Du Fu050100150200言Expected 145, actual 3闲Expected 194, actual 2海棠Expected 1.7, actual 0
Actual counts of a few characters (and one word) in Du Fu’s poems, against the counts expected at the average rate of the “Complete Tang Poems.”

In old China there was a rule of name taboo: in speech and in writing, you avoided the names of your elders. Du Fu’s avoidance of his father’s name is well known. His father was called Du Xian (杜闲), and scholars noticed long ago that the character “闲” (idle) is almost nowhere to be found in Du Fu’s poems; the few exceptions have been traced to errors made in later copying and printing. The data agree: at the average rate it should appear about 190 times, but it appears only twice.

“言” is a different matter. Tang law had a rule that “a two-character name is not tabooed one character at a time”: when an elder’s name had two characters, using just one of them was no offense. In other words, Du Fu had no need to avoid “言.” Yet in his poems, “言” all but vanishes.

Tang people were often stricter about family taboos than the law required. Because Li He’s father was named Jinsu (晋肃), some said Li He should not sit for the jinshi (进士) examination, and Han Yu wrote a whole essay, “On Taboo Names,” in his defense, citing precisely this rule about two-character names.

Was Du Fu avoiding his grandfather’s name? The data can offer a clue, but not a verdict. I haven’t found any specific discussion of this in the usual sources, and I would welcome corrections from readers who know the field.

For contrast, there is a widely repeated story that Du Fu’s mother was named Haitang (海棠, crabapple blossom), so he never wrote about crabapples. The data find nothing to support it: “海棠” appears only 41 times in the whole “Complete Tang Poems,” and for the amount Du Fu wrote, he would only be expected to mention it once or twice. Not writing it is nothing unusual.

Before calling a character “rare,” ask how often it ought to appear.

07
BAI JUYI

Could an Old Woman Really Understand Bai Juyi?

A Song-dynasty book of jottings tells a story: whenever Bai Juyi wrote a poem, he would first read it to an old woman.

Whenever Bai Letian wrote a poem, he had an old woman explain it, asking, “Do you understand?” If she said she did, he wrote it down; if not, he changed it.— Huihong, “Night Talks in the Cold Studio”

The story may not be true; it was doubted long ago. But if it were right, Bai Juyi’s characters should be the most “ordinary.” I took the 1,000 most common characters in the “Complete Tang Poems” as the “common characters,” worked out what share of each poet’s characters fall outside that set, and compared the 61 poets with at least 200 poems to their name.

10%15%20%25%30%Median of 61 poets: 17.0%Zhao Gu11.0%Liu C.12.5%Zhang Ji14.1%Li Bai16.2%Wang Wei16.7%Meng H.17.4%Bai Juyi17.9%Liu Yuxi18.5%Du Mu18.8%Li S.20.1%Du Fu21.8%Li He22.1%Yuan Zhen23.7%Lu G.24.0%Pi Rixiu25.4%Han Yu28.2%
Share of characters outside the 1,000 most common (16 of the poets; the horizontal lines are 95% intervals from resampling poems).

The result is a bit of a surprise. Bai Juyi comes in at 17.9%, 36th of the 61 poets: slightly “rarer” than the median (17.0%), and rarer than Li Bai (16.2%) too. The most “ordinary” is Zhao Gu (11.0%), and the “rarest” is Han Yu (28.2%), followed closely by Pi Rixiu, Lu Guimeng, Yuan Zhen, Li He and Du Fu.

But the share alone can mislead. Take the characters each poet uses most outside the common set, and the differences become clear:

Bai Juyi17.9% rare
劝慵勿睡富彼妻味饱司退酌
Mostly words of daily life
Han Yu28.2% rare
汝余视谓固罪杂腾施颠母鬼
Often the tone of classical prose
Li He22.1% rare
母蒲铜团蛾踏尾刺碎脉娥虫
Dark, chilly images
Du Fu21.8% rare
汝戈巫甲惨坤冬适焉黑涛奔
War and the Wu Gorge
For four poets, the 12 characters each uses most outside the 1,000 common ones, in order of frequency, with no picking and choosing.

Bai Juyi’s “rare characters” are 睡 (sleep), 饱 (full), 味 (taste), 妻 (wife), 慵 (lazy) and 酌 (pour wine): eating and sleeping, drinking and lazing about, all words for getting through the day. They are rare in poetry but used every day in life, so of course an old woman could understand them. Han Yu’s are 汝 (you), 余 (I), 谓 (say) and 固 (indeed): words from prose essays, which bears out what later critics said of him, that he “wrote poetry as prose.” Li He’s are 铜 (bronze), 蛾 (moth), 碎 (shattered) and 虫 (insect): gloomy, cold and eerie. Du Fu’s are 戈 (spear), 甲 (armor), 巫 (Wu, as in the Wu Gorge) and 涛 (waves): war and upheaval, and a man stranded in the Wu Gorge.

So “even an old woman could understand” measures the characters common in daily life, not the ones common in poetry. Measured with poetry’s ruler, Bai Juyi is not the plainest poet.

Bai Juyi’s “rare” characters are rare because he wrote everyday life into his poems.

08
MY TAKE

Numbers Can Count, but Not Read

Having counted these forty-odd thousand poems, here are a few thoughts of my own.

1. What data does best is see what is “not there.”

When people read poems, they read the characters on the page; a character that was never written is hard to notice. You could read Du Fu a thousand times and still not notice that “言” is missing; a computer counts once and sees it.

2. But numbers can’t read poetry.

“不” is the most common character in Tang poetry, yet on its own there is no poetry in it at all. What makes “不知” and “不见” moving is what comes after them: “不知何处是他乡” (“I cannot tell which land is foreign”), “不见长安见尘雾” (“Chang’an is nowhere to be seen, only dust and haze”).

3. The ruler decides the conclusion.

Measured by “characters common in poetry,” Bai Juyi is not plain; switch to “characters common in daily life,” and the conclusion may well flip. Every statistic begins by choosing a ruler. Before you look at the conclusion, look at the ruler.

I wrote before about reason and emotion. This issue is the “reason” half: data can tell us what the poets wrote and how much of it; as for why it moves us, that is still for each reader to feel.

09
CODA

In closing

Forty-odd thousand poems, more than two and a half million characters, and at the end of the count, the character in first place is “不” (not), and the question asked most often is “何处” (where).

“What poets write about most
is what they do not have.
✎ Let’s talk in the comments

Who is your favorite Tang poet? Tell me in the comments, and next time I’ll count which characters they favored most.

Appendix: the 50 most common characters in the “Complete Tang Poems”
不26,007
人20,659
山15,892
无15,599
风15,392
一15,017
日14,731
云12,560
有12,514
来12,119
天12,025
何12,020
中11,242
时11,178
花11,140
上10,940
春10,908
水10,908
月10,791
相9,534
年9,404
为9,379
生9,295
君9,190
长9,018
心8,936
自8,909
如8,631
知8,545
白8,368
归8,217
此8,011
秋7,945
行7,828
见7,736
去7,671
清7,455
夜7,370
在7,303
江7,291
空7,286
得7,225
下7,199
高7,123
未6,892
明6,683
多6,533
金6,521
门6,484
青6,408
How I counted
  • Data: the “Complete Tang Poems” section of the open-source project chinese-poetry (GitHub, MIT license, October 2023 version).
  • Scope: only the first 49,942 entries of the dataset, which correspond to the 900 volumes of the Kangxi “Complete Tang Poems” (including the ci lyrics in the final volumes). Supplementary material appended later in the dataset, such as “Lost Tang Poems” (compiled by the Japanese scholar Ichikawa Kansai) and the modern “Supplement to the Complete Tang Poems,” is not included.
  • Cleaning: editorial notes and variant readings in brackets are removed; missing characters and makeshift composite characters are not counted; variant forms with the same meaning are merged (such as “回” and “迴,” or “峰” and “峯”), while forms with different meanings are counted separately (such as “雲,” cloud, and “云,” to say).
  • Duplicates: the 905 repeated poems with identical text are counted once, leaving 49,037 poems. When comparing poets, each poem is still counted under the name the “Complete Tang Poems” assigns it.
  • Units: character frequencies are numbers of occurrences; “x% of poems” counts poems where a character appears at least once; each poem in a sequence is counted separately (merging sequences into single poems would give about 42,400). Counts use the original traditional characters, converted to simplified for display.
  • Method: poets’ preferences are measured with the log-odds ratio with an informative prior (Monroe et al., 2008); changing the prior strength to 2,000 or 50,000 leaves the top ten nearly unchanged. “Common characters” are the 1,000 most frequent in the “Complete Tang Poems”; the 95% intervals for the shares come from resampling poems 1,000 times.
  • Limitations: a character is not a word; colors and numbers are counted by character; the “Complete Tang Poems” itself contains misattributed, duplicated and doubtfully attributed poems; many similar character-frequency counts already exist online and in scholarship, and their overall conclusions are much the same.
References
  1. chinese-poetry: a database of classical Chinese poetry (GitHub, MIT license), “Complete Tang Poems” section
  2. Peng Dingqiu et al., eds., “Complete Tang Poems” (Zhonghua Book Company, 1960 typeset edition, with Ichikawa Kansai’s “Lost Tang Poems” appended); Chen Shangjun, ed., “Supplement to the Complete Tang Poems” (Zhonghua Book Company, 1992)
  3. B. L. Monroe, M. P. Colaresi & K. M. Quinn, Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict, Political Analysis 16(4), 2008
  4. Chao-Lin Liu et al., Color Aesthetics and Social Networks in Complete Tang Poems, PACLIC 29, 2015
  5. Han Yu, “On Taboo Names”; Huihong, “Night Talks in the Cold Studio,” chapter 1; “Book of Rites,” “Quli,” part 1
  6. Li Bai, “Song of a Traveler”; Bai Juyi, “Song of Everlasting Sorrow”
Note: all charts in this article were computed and drawn by the author from the data above.
犹剑在心
北极甜虾(剑心犹在!)
— The End —

← All essays