Foundations
The architecture of intelligence. Understand Karpathy's world before you enter it.
1.1 — Who Is Andrej Karpathy?
Andrej Karpathy is one of the most influential figures in modern AI. He studied under Andrew Ng at Stanford, became Tesla's Director of AI (leading Autopilot's neural network vision), and was a founding member of OpenAI.
But his real gift isn't building models — it's teaching people how to think about them.
His YouTube lectures on neural networks, transformers, and GPT have been watched millions of times. His "Let's Build GPT From Scratch" video is considered the single best introduction to how large language models actually work.
In 2026, he pioneered the autoresearch concept: a system where an AI agent autonomously runs machine learning experiments, iterates on code, and discovers optimal architectures — all while you sleep.
"The shift: from implementing experiments to designing experiments."
— Andrej Karpathy
Why this matters to you: Karpathy's approach democratizes ML research. You don't need a PhD or a cluster of H100s. You need a Mac, some curiosity, and a willingness to let the machine explore.
1.2 — Transformer Architecture Basics
Every autoresearch experiment trains a transformer — the architecture behind GPT, Claude, and every modern LLM. You don't need to understand every math detail, but you need the mental model.
The Core Idea
Transformers process text by asking one question at every step: "Which other tokens in this sequence are relevant to understanding THIS token?"
That's called attention. It's the key innovation.
The Components
Converts text → numbers. "hello" → [31373]. The model only sees numbers.
Each token becomes a dense vector (a list of numbers) that captures meaning.
Each token looks at all other tokens to decide what matters. The magic ingredient.
After attention, each token is processed independently through a small neural network.
Depth = Number of Layers
A transformer with depth 4 has 4 identical blocks stacked on top of each other. Each block = attention + MLP. More layers = more capacity to learn complex patterns, but slower training.
DEPTH = 4 # Number of transformer layers
# More layers = better (if you have RAM)
# Less layers = faster training (more steps in 5 min)
Key insight: The optimal depth depends on YOUR hardware. An M4 Mini might find depth=4 optimal, while an M5 MacBook Pro with 24GB could push to depth=6 or 8.
1.3 — Key Concepts: The Numbers That Matter
Autoresearch gives you a handful of numbers after each experiment. Learn to read them and you'll understand everything.
val_bpb — The Only Metric That Matters
Bits Per Byte measures how well the model compresses text. Lower = better compression = smarter model.
Why bpb and not accuracy? Because accuracy depends on vocabulary size. bpb is universal — you can compare models with different tokenizers fairly.
Other Training Metrics
The Fixed Time Budget
This is Karpathy's genius design choice. Every experiment trains for exactly 5 minutes. No more, no less.
Why? Because it creates a fair comparison. A tiny model sees more data in 5 minutes. A big model has more capacity but fewer updates. The optimal size depends on YOUR hardware.
val_bpb: 2.226376
training_seconds: 302.6
peak_vram_mb: 11023.9
num_steps: 101
num_params_M: 11.5
depth: 4
1.4 — Checkpoint Quiz
Q1: What does val_bpb measure?
Q2: Why does Karpathy use a fixed 5-minute training budget?
Q3: What does 'depth' mean in a transformer?
Q4: What's a good val_bpb target for a small model?
The Autoresearch Loop
The infinite lab. How the machine thinks, experiments, and learns.
2.1 — The Three Sacred Files
Autoresearch runs on three files. Understanding what each one does is the key to understanding the entire system.
Downloads the training data and trains a tokenizer. You run this ONCE and forget about it.
This is where the model lives. The AI agent edits this file to run experiments.
This is YOUR file. You write high-level research strategy here. The AI reads it and generates experiments accordingly.
The relationship: prepare.py sets the stage. train.py is what gets modified. program.md is your strategic brain guiding the modifications. You control the direction; the AI controls the implementation.
2.2 — The Experiment Cycle
One experiment cycle takes about 6-7 minutes. Here's what happens in each one:
The AI reads program.md (your strategy) and the current train.py (the model). Understands the state of the research.
Based on previous results and your strategy, the AI forms a hypothesis: "If I reduce the learning rate, maybe the model converges more smoothly."
The AI edits train.py — changing one or more variables. Each change is committed to git so you can see exactly what was tried.
Run uv run train.py — the model trains for exactly 5 minutes. The AI watches the loss decrease in real-time.
Extract val_bpb from the output. Compare to the current best. Is this an improvement?
Better? Keep it (git commit). Worse? Revert it (git reset --hard). Log the result either way.
The beauty: Most experiments will fail. That's fine. The system only keeps improvements. Over 70 experiments, even a few winners compound into significant progress.
2.3 — Reading Results Like a Scientist
After an overnight run, you'll have a results.tsv file with every experiment's outcome. Here's how to read it like a scientist.
The results.tsv Format
How to Analyze Patterns
If 5 "reduce learning rate" experiments all improved, that's a signal. If depth changes all failed, your hardware might be depth-limited.
Don't look at individual experiments. Look at the BEST val_bpb over time. Is it steadily improving? Flat trend = stagnation, change your strategy.
Some improvements cost memory. If a change saves 0.05 val_bpb but uses 2GB more RAM, it might not be worth it if you're near your limit.
What Good Progress Looks Like
Red flag: If you see 20+ consecutive discards with no keeps, your program.md strategy needs rethinking. The AI is exploring dead ends.
2.4 — Checkpoint Quiz
Q1: Which file does the AI agent modify during experiments?
Q2: What happens when an experiment is WORSE than the baseline?
Q3: What file do YOU write to guide the research direction?
Q4: If you see 20+ consecutive discards in results, what does it mean?
Hands-On
Your first experiment. Set up, run, and see what happens.
3.1 — Setting Up Your Lab
Time to build your lab. Here's exactly what you need to run autoresearch on your Mac.
Prerequisites
Step 1: Install uv (Package Manager)
Step 2: Clone the MLX Fork
Step 3: Prepare the Data
Step 4: Verify MLX is Using Metal
"Attempting to allocate X bytes" error: Your eval batch size is too big for your RAM. Edit train.py and reduce FINAL_EVAL_BATCH_SIZE from 256 to 128.
Slow tok/sec (< 15,000): MLX isn't using Metal. Close other apps and check Activity Monitor. You should see ~20k+ tok/sec on M4.
3.2 — Establishing Your Baseline
Before running experiments, you need to know your starting point. This is your baseline — the score the AI will try to beat.
Run Your First Training
Understanding Your Output
Record Your Baseline
Write it down somewhere safe. You'll reference this number constantly.
| Machine | val_bpb | Notes |
|---|---|---|
| M4 Max #1 | 1.294 | AdamW, low LR |
| M4 Max #2 | 1.331 | Leaner batch, SiLU |
| Mac Mini (long) | 1.353 | Muon optimizer |
| Our M4 Mini | 2.226 | Your starting point |
The published baselines ran longer experiments or used advanced optimizers. Your overnight runs will close the gap.
3.3 — Your First Overnight Run
This is the moment. You're about to let the machine run research while you sleep. Here's exactly what to do.
Before You Start
Launch It
What to Expect
Emergency Stop
Start with a SHORT run first (30 minutes / ~5 experiments) before committing to a full overnight session. Make sure everything works before you walk away.
3.4 — Checkpoint Quiz
Q1: What command prepares the training data (run once)?
Q2: What should you see when verifying MLX uses Metal?
Q3: How do you emergency-stop an autoresearch run?
Q4: What should you do BEFORE your first overnight run?
Advanced Orchestration
The meta-game. Design experiments from intuition, not guessing.
4.1 — Writing program.md Like a Research Director
program.md is your most powerful lever. The quality of your strategy determines the quality of the AI's experiments. Most people write bad program.md files. Here's how to write great ones.
The Three Levels of program.md
This is too vague. The AI will wander randomly. You'll get maybe 1-2 lucky keeps out of 70 experiments.
Better! The AI has specific values to try and a fallback plan. You'll get 5-10 keeps.
This is elite. The AI knows the full state, has a specific hypothesis, prioritized experiments, and documented dead ends. You'll get 15-20 keeps.
The Golden Rules
4.2 — Multi-Machine Research
You have two machines: Theta-Server (M4 Mini, 16GB) and your MacBook Pro (M5, 24GB). Running research on both simultaneously doubles your progress.
Why Two Machines > One
The Two-Pronged Strategy
Squeeze every drop out of depth=4. Focus on learning rate, batch size, weight decay, optimizer tweaks. Small changes, big impact.
With 24GB, you can try depth=6, depth=8, bigger batches. Explore architecture changes that would OOM on the Mini. Different optimizer experiments.
Syncing Results
When the M5 finds a good trick with depth=6, try adapting it for the M4's depth=4. Architectural insights often transfer even when exact configs don't.
4.3 — Interpreting Results and Patterns
Raw results tell you what happened. Your job is to figure out why. This is the difference between a data collector and a scientist.
The Analysis Framework
Export your results.tsv, sort by val_bpb ascending (best first). The top 10 experiments are your gold.
Look at your top performers. What do they have in common?
Single-variable analysis misses interactions. A learning rate that's bad alone might be great combined with specific batch sizes.
Common Patterns to Watch For
First 20 experiments improve fast, then flatline. Normal — you've found the easy wins. Time for a new hypothesis.
30+ experiments stuck at 2.15, then suddenly one hits 2.08. The AI found a new regime. Check what changed.
Everything at depth=5 works, depth=6 crashes. You've found your hardware ceiling. Maximize depth=5 instead.
The "So What" Test
After analyzing, ask yourself:
Update program.md with answers to all 5 questions. This becomes your next session's starting strategy.
4.4 — Checkpoint Quiz
Q1: What makes a Level 3 program.md better than Level 1?
Q2: What's the benefit of running research on two machines simultaneously?
Q3: What should you do after every research session?
Q4: What does an "OOM Cliff" pattern tell you?
Mastery
Beyond autoresearch. Apply these principles to everything.
5.1 — Scaling Your Research Workflow
You understand the system. You can run experiments. Now let's talk about scaling your research into something that compounds over time.
The Research Portfolio
Don't run one research stream — run a portfolio. Each session builds on the last.
A running document: what you tried, what worked, what to do next. This is your lab notebook.
Every experiment's outcome. Sortable, analyzable. Your data spine.
Git tracks your strategy evolution. See how your thinking changed over time.
Distill learnings into permanent knowledge. "Depth=5 OOMs on M4 at batch 2^16" is a fact forever.
Time Investment Strategy
Scaling Beyond One Project
Autoresearch isn't just about one model. You can run parallel research streams:
5.2 — Autoresearch Beyond ML
Here's the secret: the autoresearch pattern isn't limited to ML training. It's a meta-framework for autonomous iteration. Anywhere you have: measurable output + modifiable code + a goal to optimize — autoresearch works.
The Pattern
Domains Where This Works
Autonomously optimize API response times, database queries, or build pipelines. Metric: latency/throughput. Change: code modifications.
Optimize ETL processes, batch sizes, parallelism settings. Metric: processing time. Change: configuration parameters.
Train game-playing agents through self-play. Metric: win rate. Change: strategy parameters, neural network architecture.
Backtest and iterate on trading strategies. Metric: Sharpe ratio, returns. Change: entry/exit rules, position sizing. (Use paper trading!)
Optimize server configs, container orchestration, resource allocation. Metric: cost-performance ratio. Change: infrastructure parameters.
Karpathy's genius wasn't inventing the concept of optimization. It was designing a system where the human sets direction and the AI handles execution. That separation — strategy vs. implementation — works everywhere.
5.3 — The Future of AI-Assisted Research
You've learned the system. Now zoom out. Where is all of this going?
The Bigger Picture
Karpathy's autoresearch is a glimpse of a broader shift: AI agents doing science. Not replacing scientists — augmenting them. Handling the tedious implementation work so humans can focus on creative direction.
What Karpathy Is Teaching
Your Next Steps
The lab is yours.
Go build something impossible.
5.4 — Final Assessment
Q1: What's the key principle that makes autoresearch work beyond ML training?
Q2: What's the recommended weekly research cadence?
Q3: What's Karpathy's key teaching about the agent era?
Q4: What's the ultimate goal of mastering autoresearch?
You've Mastered the Machine
You now understand Karpathy's autoresearch from architecture to execution. The lab is yours. Go build something impossible.