Seeing inside the
mind of AI

01Read the logprobs, not the answer

The text a model gives you is the tip of the iceberg. Ask the API for token logprobs and you can see where it was confident versus where it basically flipped a coin. A fluent sentence built on 51% tokens is a guess wearing a suit.

Confidence ≠ correctness. But low confidence is a great place to look for hallucinations.

#interpretability

02Make it think before it answers

Put the reasoning before the conclusion, never after. If you ask for the answer first and the explanation second, the explanation is just a story it invents to justify a choice it already made. Order matters more than wording.

#prompting

03BertViz for watching attention move

An open-source visualizer that shows which tokens each attention head is looking at, layer by layer. You start to notice heads that only track quotation marks, or ones that follow subject-verb agreement across a whole paragraph.

Most heads are boring. The interesting 5% are worth the dig.

#tooling

04Tell it what NOT to do, sparingly

Negative instructions are weak anchors. 'Don't mention X' often makes X more likely because you just put X in the context. Prefer describing the thing you DO want. Reserve negatives for hard safety rails, not style.

#prompting

05Probe for a concept with a linear classifier

Freeze the model, grab the hidden activations for a pile of examples, and train a tiny linear probe to predict some property (is this text about money? is it past tense?). If a simple line separates them, the concept is already represented in there, plain as day.

Cheap, fast, and weirdly revealing.

#interpretability

06Temperature is a creativity dial, not a quality dial

Turning it up doesn't make answers smarter, it makes them less predictable. For extraction, classification, and anything with a right answer, keep it near zero. For brainstorming, let it wander. People mix these up constantly.

#prompting

07Sparse autoencoders to untangle neurons

Individual neurons are polysemantic, one neuron fires for 'the Golden Gate Bridge' and also 'feeling trapped' and also semicolons. Train a sparse autoencoder on the activations and you pull those mixed signals apart into cleaner, single-meaning features.

This is roughly how the 'Golden Gate Claude' demo worked.

#interpretability

08Keep a tokenizer open in a tab

Half of weird model behavior is a tokenization story. ' apple' and 'apple' are different tokens. Numbers split in unintuitive ways. Pasting your prompt into a tokenizer viewer takes ten seconds and explains a surprising number of bugs.

#tooling

09Sample many, keep the majority

For reasoning tasks, generate the same answer five or ten times at moderate temperature and take the most common result. A single greedy answer is fragile; the consensus across samples is far more reliable. Slow, but it works.

#reliability

10Mind the middle of long contexts

Models attend strongly to the beginning and end of a long prompt and get foggy in the middle. Put the instructions and the most important facts at the edges. If something keeps getting ignored, it's probably buried in the soft center.

Sometimes called 'lost in the middle'.

#prompting

11Activation patching to find the cause

Run the model twice, once on a clean prompt and once on a corrupted one, then copy a single activation from one run into the other. If the output flips, you just found a component that carries the behavior. It's causal, not just correlational.

The closest thing in this field to a controlled experiment.

#interpretability

12Your examples teach format louder than content

In few-shot prompts the model copies the shape of your examples first: the punctuation, the casing, the length, the label words. Get the format dead consistent and the task often solves itself. Sloppy examples teach sloppiness.

#prompting

13The logit lens: decode the middle layers

Take the hidden state from an intermediate layer and run it through the model's output head early. You get the model's 'best guess so far' at that depth. Watching the prediction sharpen layer by layer is like a time-lapse of a thought forming.

#interpretability