Two tiny transformers

July 17, 2026 · ml, transformers

Since last week, I have been going through Karpathy’s “Let’s build GPT” If you are interested in understanding transformers, I implore you to go through this video. If you lack prerequisite ML knowledge then it may be more productive to spend some time understanding the fundamentals (maybe a week?) and then go through the video.

I now understand Attention, KVQ, and the Transformemr better. I do need to read the Attention paper in full to understand some the things at a deeper level such as why Scaled Dot Product Attention is needed. I found the overall experience to be much more like Alchemy than Engineering but it was very fun.

Example

The tiny transformer model initially outputs garbeled text as follow:

l me son?dEiP ,ydh,cGPkaF ?SV
kkdQOi'

After some training it turns into this:

ISABELLA:
Let, and wilts; let see of you'l come thou,
Which which'd computs fallain, unly mrobe noisur

Training Arithmetic

This still does not make any sense and for now I attribute it to the fact that the model is very small and the tokenization is per character.

I was also able to teach the model to do Addition and Multiplication. You have to create the training data which includes Inputs and Targets. Most of my effort went into designing the training data.

Addition

The first approach is to create data as follows:
Input: “24+29” Target: “53”

But this is not the most effective because it hides an implicit step of carrying i.e. 9+4 = 13 and you have to carry the 1. To make this easier to understand you can write itin reverse as follows: Input: “42+92” Target: “53”

This way the model first adds the digits in the One’s place i.e. 9 and 4 and then adds the digits in the Ten’s place i.e. 4 and 9 and adds the carry.

Notebooks: nanogpt.ipynb and addition_lm.ipynb. multiplication_lm.ipynb