glassllmLab
DocumentationHow it works

Explore the model from within

Free mode

Write a beginning. Follow each choice, one token at a time.

EN model·In your browser

Start with a sentence

Input

Use “Step once” to inspect a single prediction.

Generated text

0 tokens
Tokens BPEeach box shows its token and ID

Start generation to see how your prompt is split.

The token’s path

Explore the network
01TokensIDs (n,)
02EmbeddingE[ids] → (n, 128)
03Blocksattention + FFN
1Block 1
2Block 2
3Block 3
4Block 4
04Logits(V,)
05Samplingsoftmax / T

Attention

Context

Where the last token looked.

layer 4 · head 0

After the first step, inspect each context token’s weight.

Next possibilities

Top 6

Candidates for continuing your prompt.

softmax(logits / T)

Generate a token to compare the six highest probabilities.

From text to the next token

Inside the complete network

4 blocks · 128 dimensions · 4 Q / 2 K/V heads · 128-token context

Generation architecture
100%

Enable before generating or replaying. Each step shows its explanation and actual values; pause to read at your own pace.

Block 1Causal attention → SwiGLU · two residual pathsBlock 2Causal attention → SwiGLU · two residual pathsBlock 3Causal attention → SwiGLU · two residual pathsBlock 4Causal attention → SwiGLU · two residual paths X0t−1VHX1t−1VHX2t−1VHX3t−1VHz[−1] / TThe same matrix E · tied weightsNew token → update context → next forward pass Initial text Prompt stringInitial textPromptstringBPE tokenizer Text → integer ids 1024 vocabulary tokensBPE tokenizerText → integer ids1024 vocabulary tokensToken context ids + new id Window: up to 128Token contextids + new idWindow: up to 128Token embedding X₀ = E[ids] n × 128Token embeddingX₀ = E[ids]n × 128Shared weights · E Embedding and output use E 1024 × 128Shared weights · EEmbedding and output use E1024 × 128One Transformer forward pass n = new tokens; m = current + cached keys 4 blocks · d = 128 · 4 Q / 2 KV · dₖ = 32One Transformer forward passn = new tokens; m = current + cached keys4 blocks · d = 128 · 4 Q / 2 KV · dₖ = 32RMSNorm · attention X̂ = X0 / √(mean(X0²) + ε) · γ₁ n × 128RMSNorm · attentionX̂ = X0 / √(mean(X0²) + ε) · γ₁n × 128Q · query Q = X̂ · W_Q 4 × n × 32Q · queryQ = X̂ · W_Q4 × n × 32K · key K = X̂ · W_K 2 × n × 32K · keyK = X̂ · W_K2 × n × 32V · value V = X̂ · W_V 2 × n × 32V · valueV = X̂ · W_V2 × n × 32RoPE · Q Q′ = rotate(Q, pos) 4 × n × 32RoPE · QQ′ = rotate(Q, pos)4 × n × 32RoPE · K K′ = rotate(K, pos) 2 × n × 32RoPE · KK′ = rotate(K, pos)2 × n × 32K/V memory · cache Append K′ and V to previous tokens 2 × m × 32 · K / VK/V memory · cacheAppend K′ and V to previous tokens2 × m × 32 · K / VExpand heads · GQA repeat_kv(K, 2) · repeat_kv(V, 2) 2 → 4 headsExpand heads · GQArepeat_kv(K, 2) · repeat_kv(V, 2)2 → 4 headsQ × K compatibility S = Q′ · K′ᵀ / √32 4 × n × mQ × K compatibilityS = Q′ · K′ᵀ / √324 × n × mCausal mask Sᵢⱼ = −∞ if j > positionᵢ 4 × n × mCausal maskSᵢⱼ = −∞ if j > positionᵢ4 × n × mAttention weights A = softmax(S) 4 × n × mAttention weightsA = softmax(S)4 × n × mMix the values Y_h = A · V 4 × n × 32Mix the valuesY_h = A · V4 × n × 32Concatenate and project O = Concat(Y₁ … Y_H) · W_O n × 128Concatenate and projectO = Concat(Y₁ … Y_H) · W_On × 128Add the residual path H = X0 + O n × 128Add the residual pathH = X0 + On × 128RMSNorm · FFN Ĥ = H / √(mean(H²) + ε) · γ₂ n × 128RMSNorm · FFNĤ = H / √(mean(H²) + ε) · γ₂n × 128Gate projection G = Ĥ · W₁ n × 341Gate projectionG = Ĥ · W₁n × 341Up projection U = Ĥ · W₃ n × 341Up projectionU = Ĥ · W₃n × 341SwiGLU activation SiLU(G) ⊙ U n × 341SwiGLU activationSiLU(G) ⊙ Un × 341Down projection F = (SiLU(G) ⊙ U) · W₂ n × 128Down projectionF = (SiLU(G) ⊙ U) · W₂n × 128Add the residual path X1 = H + F n × 128Add the residual pathX1 = H + Fn × 128RMSNorm · attention X̂ = X1 / √(mean(X1²) + ε) · γ₁ n × 128RMSNorm · attentionX̂ = X1 / √(mean(X1²) + ε) · γ₁n × 128Q · query Q = X̂ · W_Q 4 × n × 32Q · queryQ = X̂ · W_Q4 × n × 32K · key K = X̂ · W_K 2 × n × 32K · keyK = X̂ · W_K2 × n × 32V · value V = X̂ · W_V 2 × n × 32V · valueV = X̂ · W_V2 × n × 32RoPE · Q Q′ = rotate(Q, pos) 4 × n × 32RoPE · QQ′ = rotate(Q, pos)4 × n × 32RoPE · K K′ = rotate(K, pos) 2 × n × 32RoPE · KK′ = rotate(K, pos)2 × n × 32K/V memory · cache Append K′ and V to previous tokens 2 × m × 32 · K / VK/V memory · cacheAppend K′ and V to previous tokens2 × m × 32 · K / VExpand heads · GQA repeat_kv(K, 2) · repeat_kv(V, 2) 2 → 4 headsExpand heads · GQArepeat_kv(K, 2) · repeat_kv(V, 2)2 → 4 headsQ × K compatibility S = Q′ · K′ᵀ / √32 4 × n × mQ × K compatibilityS = Q′ · K′ᵀ / √324 × n × mCausal mask Sᵢⱼ = −∞ if j > positionᵢ 4 × n × mCausal maskSᵢⱼ = −∞ if j > positionᵢ4 × n × mAttention weights A = softmax(S) 4 × n × mAttention weightsA = softmax(S)4 × n × mMix the values Y_h = A · V 4 × n × 32Mix the valuesY_h = A · V4 × n × 32Concatenate and project O = Concat(Y₁ … Y_H) · W_O n × 128Concatenate and projectO = Concat(Y₁ … Y_H) · W_On × 128Add the residual path H = X1 + O n × 128Add the residual pathH = X1 + On × 128RMSNorm · FFN Ĥ = H / √(mean(H²) + ε) · γ₂ n × 128RMSNorm · FFNĤ = H / √(mean(H²) + ε) · γ₂n × 128Gate projection G = Ĥ · W₁ n × 341Gate projectionG = Ĥ · W₁n × 341Up projection U = Ĥ · W₃ n × 341Up projectionU = Ĥ · W₃n × 341SwiGLU activation SiLU(G) ⊙ U n × 341SwiGLU activationSiLU(G) ⊙ Un × 341Down projection F = (SiLU(G) ⊙ U) · W₂ n × 128Down projectionF = (SiLU(G) ⊙ U) · W₂n × 128Add the residual path X2 = H + F n × 128Add the residual pathX2 = H + Fn × 128RMSNorm · attention X̂ = X2 / √(mean(X2²) + ε) · γ₁ n × 128RMSNorm · attentionX̂ = X2 / √(mean(X2²) + ε) · γ₁n × 128Q · query Q = X̂ · W_Q 4 × n × 32Q · queryQ = X̂ · W_Q4 × n × 32K · key K = X̂ · W_K 2 × n × 32K · keyK = X̂ · W_K2 × n × 32V · value V = X̂ · W_V 2 × n × 32V · valueV = X̂ · W_V2 × n × 32RoPE · Q Q′ = rotate(Q, pos) 4 × n × 32RoPE · QQ′ = rotate(Q, pos)4 × n × 32RoPE · K K′ = rotate(K, pos) 2 × n × 32RoPE · KK′ = rotate(K, pos)2 × n × 32K/V memory · cache Append K′ and V to previous tokens 2 × m × 32 · K / VK/V memory · cacheAppend K′ and V to previous tokens2 × m × 32 · K / VExpand heads · GQA repeat_kv(K, 2) · repeat_kv(V, 2) 2 → 4 headsExpand heads · GQArepeat_kv(K, 2) · repeat_kv(V, 2)2 → 4 headsQ × K compatibility S = Q′ · K′ᵀ / √32 4 × n × mQ × K compatibilityS = Q′ · K′ᵀ / √324 × n × mCausal mask Sᵢⱼ = −∞ if j > positionᵢ 4 × n × mCausal maskSᵢⱼ = −∞ if j > positionᵢ4 × n × mAttention weights A = softmax(S) 4 × n × mAttention weightsA = softmax(S)4 × n × mMix the values Y_h = A · V 4 × n × 32Mix the valuesY_h = A · V4 × n × 32Concatenate and project O = Concat(Y₁ … Y_H) · W_O n × 128Concatenate and projectO = Concat(Y₁ … Y_H) · W_On × 128Add the residual path H = X2 + O n × 128Add the residual pathH = X2 + On × 128RMSNorm · FFN Ĥ = H / √(mean(H²) + ε) · γ₂ n × 128RMSNorm · FFNĤ = H / √(mean(H²) + ε) · γ₂n × 128Gate projection G = Ĥ · W₁ n × 341Gate projectionG = Ĥ · W₁n × 341Up projection U = Ĥ · W₃ n × 341Up projectionU = Ĥ · W₃n × 341SwiGLU activation SiLU(G) ⊙ U n × 341SwiGLU activationSiLU(G) ⊙ Un × 341Down projection F = (SiLU(G) ⊙ U) · W₂ n × 128Down projectionF = (SiLU(G) ⊙ U) · W₂n × 128Add the residual path X3 = H + F n × 128Add the residual pathX3 = H + Fn × 128RMSNorm · attention X̂ = X3 / √(mean(X3²) + ε) · γ₁ n × 128RMSNorm · attentionX̂ = X3 / √(mean(X3²) + ε) · γ₁n × 128Q · query Q = X̂ · W_Q 4 × n × 32Q · queryQ = X̂ · W_Q4 × n × 32K · key K = X̂ · W_K 2 × n × 32K · keyK = X̂ · W_K2 × n × 32V · value V = X̂ · W_V 2 × n × 32V · valueV = X̂ · W_V2 × n × 32RoPE · Q Q′ = rotate(Q, pos) 4 × n × 32RoPE · QQ′ = rotate(Q, pos)4 × n × 32RoPE · K K′ = rotate(K, pos) 2 × n × 32RoPE · KK′ = rotate(K, pos)2 × n × 32K/V memory · cache Append K′ and V to previous tokens 2 × m × 32 · K / VK/V memory · cacheAppend K′ and V to previous tokens2 × m × 32 · K / VExpand heads · GQA repeat_kv(K, 2) · repeat_kv(V, 2) 2 → 4 headsExpand heads · GQArepeat_kv(K, 2) · repeat_kv(V, 2)2 → 4 headsQ × K compatibility S = Q′ · K′ᵀ / √32 4 × n × mQ × K compatibilityS = Q′ · K′ᵀ / √324 × n × mCausal mask Sᵢⱼ = −∞ if j > positionᵢ 4 × n × mCausal maskSᵢⱼ = −∞ if j > positionᵢ4 × n × mAttention weights A = softmax(S) 4 × n × mAttention weightsA = softmax(S)4 × n × mMix the values Y_h = A · V 4 × n × 32Mix the valuesY_h = A · V4 × n × 32Concatenate and project O = Concat(Y₁ … Y_H) · W_O n × 128Concatenate and projectO = Concat(Y₁ … Y_H) · W_On × 128Add the residual path H = X3 + O n × 128Add the residual pathH = X3 + On × 128RMSNorm · FFN Ĥ = H / √(mean(H²) + ε) · γ₂ n × 128RMSNorm · FFNĤ = H / √(mean(H²) + ε) · γ₂n × 128Gate projection G = Ĥ · W₁ n × 341Gate projectionG = Ĥ · W₁n × 341Up projection U = Ĥ · W₃ n × 341Up projectionU = Ĥ · W₃n × 341SwiGLU activation SiLU(G) ⊙ U n × 341SwiGLU activationSiLU(G) ⊙ Un × 341Down projection F = (SiLU(G) ⊙ U) · W₂ n × 128Down projectionF = (SiLU(G) ⊙ U) · W₂n × 128Add the residual path X4 = H + F n × 128Add the residual pathX4 = H + Fn × 128RMSNorm · final h = RMSNorm(X4) n × 128RMSNorm · finalh = RMSNorm(X4)n × 128Project to the vocabulary z = h · Eᵀ n × 1024Project to the vocabularyz = h · Eᵀn × 1024Next-token probabilities p = softmax(z[−1] / T) 1024Next-token probabilitiesp = softmax(z[−1] / T)1024Choose the next token Sample ~ p or argmax(p) 1 id → textChoose the next tokenSample ~ p or argmax(p)1 id → text
CalculationsResidualCache K/VE weightsCurrent step

Select a block to zoom in. Drag the background to pan and click operations to inspect them.

Complete map · waiting for the first predictionSlow motion slows the visualization of actual calculations recorded by the model.
How glassllm works

glassllm is a teaching implementation of a Transformer. Instead of hiding its calculations behind an API, it lets you follow what happens at every generation step.

The free lab follows the generation loop in real time. Set the temperature and sampling mode, step through one token at a time, and inspect attention and probabilities.

A chapter path connecting equations, code, and values is planned for a later phase.