The component that converts text to and from the integer token IDs an LLM actually consumes.
LLMs
BACKLOG · WORK IN PROGRESS
This term is being written.
The metadata and shape of this page are stable, but the body content
isn't ready yet. We'll publish it once it meets the bar of teaching
something new with worked examples and real tools.
