Count Unique Tokens
Given a sequence of 10 tokens drawn from a vocabulary of 10 symbols, predict the number of distinct symbols. 2-layer attention-only transformer, no positional embeddings, 9,600 parameters. Reverse-engineer the algorithm it learned.
The starter notebook loads the pre-trained weights from HuggingFace and walks you through basic inference and attention visualization. Open it in Colab, save a copy to your own Drive, and start exploring.
Submit a link to a Colab notebook with your findings. Your notebook should be clean and easy to follow — use markdown cells to explain your reasoning clearly, include well-labeled plots to support your claims, and walk the reader through the algorithm you found. Think of it as a presentation of your results, not scratch work.
A good workflow: do your rough exploration in a working notebook, then create a fresh notebook for your submission where you present things clearly and concisely.
Deadline: May 31, 2026 (anywhere on Earth)