About
Deeptext builds efficient, explainable language models, and explains how they work along the way.
This blog is where that explaining happens: one post a week on how language models and NLP work under the hood, from first principles and with code.
How to read the soundings
Every post opens with a figure drawn from its own words. One stroke hangs from the surface for each word, in reading order, the way a depth sounding hangs from a ship. Short posts keep every word; long ones keep the deepest word of each stretch.
The depth comes from a small language model. When a post is saved, a character bigram model is trained on that post alone. It learns how likely each letter is to follow the previous one, then measures how many bits it needs to encode each word:
Words the model finds easy stay near the surface. Long words with unusual letter patterns go deep, and those are often the words a post is really about. The deepest ones are named in the figure. On a post, move over the strokes to read any word and its cost in bits.