attogpt: A GPT on Punch Cards

How Small Can a GPT Get?

In 2024, Andrej Karpathy released microgpt, a 240-line GPT implementation. Other versions kept shrinking it while preserving a working model.

attogpt cuts the original line count by 87.1%. Its 31 lines and 3,117 characters still include autograd, multi-head attention, training, and inference.

By the Numbers

What It Actually Does

The implementation includes:

• Autograd Engine: Automatic differentiation with backward pass
• Transformer Architecture: Multi-head self-attention mechanism
• Training Loop: Adam optimizer with bias correction and learning rate decay
• Text Generation: Inference with temperature-controlled sampling

After training on a dataset of names for 1,000 steps, it generates plausible new names:

sample  1: kamon
sample  2: ann
sample  3: karai
sample  4: jaire
sample  5: vialan
sample  6: karia
sample  7: yeran
sample  8: anna
sample  9: areli
sample 10: kaina

The Punch Card Format

I also encoded the program as IBM-style punch cards, using the physical 80-column by 12-row format common in early computing.

Each Base64 character maps to a 12-bit punch pattern. This is a modern encoding, not period-correct Hollerith code, but it is reversible and fits the network on 52 cards.

Technical Details

Encoding Approach

The encoding is fully reversible:

1. Base64 Conversion: Python code → UTF-8 bytes → Base64 string
2. Binary Punch Mapping: Each Base64 character → unique 12-bit binary pattern
3. Card Layout: 80 characters per card, 12 possible punch positions per column
4. Decoding: Punch patterns → Base64 → UTF-8 → working Python code

Historical Note

The cards use modern binary encoding rather than 1960s Hollerith code because:

• Original IBM 026/029 keypunch machines only supported UPPERCASE characters
• Python requires mixed-case (keywords, variables, etc.)
• The modern encoding can represent every character and decode without loss

The cards are a size comparison, not a period-accurate implementation.

Key Compression Techniques

• Lambda Functions: Replace method definitions with inline lambdas
• Dictionary State: Flatten nested parameter structure
• Expression Folding: Combine multiple operations into single statements
• Variable Reuse: Aggressive reuse of variable names

The compressed version keeps the same functionality as microgpt and produces comparable samples.

Why Punch Cards?

Punch cards make the program's size tangible. A single card held 80 characters, and a drawer of roughly 2,000 cards held about 156 KB.

At 52 cards, attogpt takes up a small part of that drawer. The whole training and generation program fits in one hand.