🧩 词元：AI 眼里的文字积木🧩 Token: the Building Blocks of Text

上一片里，编码器和解码器一唱一和，把整句话传来传去。可它们传的到底是什么？句子进门之前，会先被拆成一块块“文字积木”——那才是模型真正读的东西。 Last leaf, the encoder and the decoder passed whole sentences back and forth like two messengers. But what exactly do they pass? Before a sentence enters the model, it gets chopped into little “text blocks” — and those blocks are what the model truly reads.

第 1 步：先切块，再进门Step 1: Chopped into blocks first

那模型眼里的“一句话”，到底长什么样？So what does a “sentence” actually look like to the model?

一句话由彩色积木拼成，小机器人在旁边 — 在模型眼里，一段文字像一盒拼好的乐高：连起来是一句话，拆开是一小块一小块的词元（token，也就是“文字积木”）。它不按“字”读，也不按“词”读，只认这盒积木——每块积木在盒里都有自己的编号。 To the model, a piece of text is like a box of Lego: snapped together it reads as a sentence; pulled apart it is a row of tokens — text blocks. It doesn’t read letter by letter, nor word by word — it reads blocks. And every block has its own number in the box.

为什么有的词整块留下，有的却被切小？Why do some words stay whole while others get chopped small?

第 2 步：常见的整块，生僻的拆开Step 2: Common words stay whole, rare ones split

playing 拆成 play+ing，tokenization 拆成 token+ization — 模型的积木盒里只有几万到十几万块，这叫词表。想给每个词都备一块，盒子根本放不下——那就改备“零件”：play 和 ing 盒里都有，playing 就各拿一块拼出来；生僻的 tokenization 拆成 token + ization，照样拼得出。再新的词，都能用盒里的积木拼出来。 The model’s block box holds only tens of thousands of blocks — that’s the vocabulary. One block per possible word would never fit, so it stocks parts instead: play and ing are both in the box, so playing is built by grabbing one of each; the rare tokenization splits into token + ization and still comes out right. Even a word nobody has ever seen can be built from blocks in the box.

这跟我们的生活有什么关系？And how does this touch our daily life?

第 3 步：字数、账单，都按块算Step 3: Counts and bills, all by the block

积木计数站：模型按块计数 — 你平时数“字”，模型数“块”。输入框的字数统计、按量收费的 API 账单（API = 程序之间说话的接口），背地里数的都是词元：中文大约 1 个字 ≈ 1 块，英文大约 4 个字符 ≈ 1 块。同一句话，语言不同，块数差很多；喂进去的上下文越长，账单越厚。 You count letters; the model counts blocks. The character counter under your text box and a metered API bill (API = how programs talk to each other) both count tokens behind the scenes: in Chinese, roughly 1 character ≈ 1 block; in English, roughly 4 characters ≈ 1 block. The same sentence costs very different block counts in different languages — and the longer the context you feed, the thicker the bill.

🎮 你来切积木（30 秒）🎮 Your turn: chop the blocks (30 seconds)

道理讲完了。下面 4 组词，每组有两种切法——点一个你认为切得对的；也可以点“自动演示”看一遍。Theory done. Here are 4 words, each with two possible splits — tap the one you think is right; or hit “Auto demo” and watch.

一句话记住它：AI 读文本之前，先把它切成词元（文字积木）；常见词整块保留、生僻词拆成零件，而字数和账单，都按块来数。 Remember it in one line: before reading anything, the AI chops text into tokens (text blocks); common words stay whole, rare ones break into parts — and character counts and bills are all counted in blocks.

文本先切块（token）再进模型——一句话就是一排拼好的积木Text is chopped into tokens before entering the model — a sentence is a row of snapped-together blocks 词表只有几万到十几万块：常见词整块留，生僻词拆小块（playing → play + ing）The block box holds only tens of thousands of blocks: common words stay whole, rare ones split (playing → play + ing) 字数和 API 计费都按词元数：中文 ≈ 1 字 1 块，英文 ≈ 4 字符 1 块Character counts and API billing count tokens: ≈ 1 Chinese character or 4 English characters per block

← 浏览全部 302 个知识点← Browse all 302 topics

内容参考 OpenStax 等公开教材，多来源核对 · AI 生成、人工审核 · 发现错误欢迎指正，帮这片叶子长得更好。 Based on OpenStax and other open textbooks, cross-checked across sources · AI-generated, human-reviewed · Spotted a mistake? Tell us — help this leaf grow.