Primary sources, their connection to the code and differences between the papers and this educational Transformer.
glassllm 0.1.04 min read · estimate
Using this bibliography
The papers below explain concepts used by the engine. Each record includes a primary link, full authors, first arXiv submission year, related pages and a preprint BibTeX citation. Publication notes distinguish venues and journals when listed by the primary record. Adam, for example, has a 2014 preprint and an ICLR 2015 publication; citing the preprint year does not change the publication date.
Use related-page links to move from an idea to its executable operation. The bibliography identifies conceptual provenance; it does not mean the project reproduces every paper’s architecture, dataset or experimental results.
Mapping papers to code
Source
Used concept
Code location
Attention Is All You Need
Scaled attention and Transformer structure
nn.Attention, nn.Block, model.GlassLM
Neural Machine Translation of Rare Words with Subword Units
Subword BPE merges
tokenizer.Tokenizer.train, Tokenizer.encode
SentencePiece
Tokenization and space representation context
tokenizer.pre_tokenize, SPACE
RoFormer
Positional rotations of queries and keys
nn.rope_angles, autograd.rope
GQA
Shared K/V heads
nn.Attention, autograd.repeat_kv
Root Mean Square Layer Normalization
Root mean square normalization
nn.RMSNorm
On Layer Normalization in the Transformer Architecture
Pre-norm placement
nn.Block
Deep Residual Learning for Image Recognition
Residual-addition context
nn.Block, resid.attn and resid.ffn
GLU Variants Improve Transformer
SwiGLU gated FFN
nn.SwiGLU, autograd.silu
Using the Output Embedding to Improve Language Models
Tied input/output embedding weights
GlassLM.E, GlassLM.forward
Automatic differentiation in machine learning: a survey
Reverse mode and chain rule
autograd.Tensor.backward
Adam: A Method for Stochastic Optimization
Moments and adaptive updates
optim.AdamW.step
Decoupled Weight Decay Regularization
Decoupled weight decay
optim.AdamW.step
SGDR: Stochastic Gradient Descent with Warm Restarts
Cosine part of the schedule
optim.lr_schedule
Module prefixes in this table are relative to glassllm. The public API reference distinguishes root exports from submodule functions.
A custom combination of architecture choices
The original Vaswani et al. Transformer uses an encoder and decoder for sequence transduction. glassllm implements a small causal decoder for next-token prediction, with no encoder or cross-attention. It uses RoPE, RMSNorm, GQA and SwiGLU, choices introduced or studied in different papers.
GQA shares two K/V heads across four query heads. The project does not run the checkpoint conversion and continued-training procedure from the GQA paper. Its tokenizer implements BPE over normalized text; SentencePiece is contextual background rather than a dependency or a format-compatibility guarantee.
Normalization and residuals
The pre-norm paper analyzes LayerNorm; this code uses RMSNorm before both sublayers and at the output. The relationship informs normalization placement, without automatically transferring all theoretical conclusions between distinct operations.
The RMSNorm paper also discusses pRMSNorm; this project computes the mean square of all dimensions. With ε > 0, scale invariance is not an exact equality for every vector. The residual-network paper supplies historical vision context rather than proving that this model keeps arbitrary gradients constant.
Training and optimization
AdamW uses gradient moments and decay separate from adaptive updates. The project’s schedule combines warmup with cosine decay down to a fraction of the initial rate. It does not implement SGDR warm restarts, and its optimizer is AdamW rather than the SGD evaluated in that paper.
Autograd is implemented locally over NumPy arrays. The automatic-differentiation survey provides terminology and a methodological overview; accumulation, broadcasting and traversal details belong to the implementation described by the Tensor reference.
Citing and checking sources
Open each arXiv record to inspect versions, PDF and publication details. This page’s BibTeX entries consistently cite preprints with authors, title, year, eprint, arXiv DOI and URL. The downloadable file contains all 14 records. For academic work requiring the published version, use the corresponding proceedings or journal reference and identify the version consulted.
Foundation equations also state the actual code’s shapes and choices. Start with causal attention and loss and gradients to connect a paper to an executed numerical calculation.
Project sources
README.mddocs/equacoes.md
Related papers
Primary sources for concepts used in the project. The year identifies the first arXiv preprint version; the publication note names the venue or journal when recorded by the source.
@misc{glassllm_transformer_2017,
title = {Attention Is All You Need},
author = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},
year = {2017},
eprint = {1706.03762},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1706.03762},
url = {https://arxiv.org/abs/1706.03762}
}
@misc{glassllm_bpe_2015,
title = {Neural Machine Translation of Rare Words with Subword Units},
author = {Rico Sennrich and Barry Haddow and Alexandra Birch},
year = {2015},
eprint = {1508.07909},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1508.07909},
url = {https://arxiv.org/abs/1508.07909}
}
@misc{glassllm_sentencepiece_2018,
title = {SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing},
author = {Taku Kudo and John Richardson},
year = {2018},
eprint = {1808.06226},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1808.06226},
url = {https://arxiv.org/abs/1808.06226}
}
@misc{glassllm_rope_2021,
title = {RoFormer: Enhanced Transformer with Rotary Position Embedding},
author = {Jianlin Su and Yu Lu and Shengfeng Pan and Ahmed Murtadha and Bo Wen and Yunfeng Liu},
year = {2021},
eprint = {2104.09864},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.2104.09864},
url = {https://arxiv.org/abs/2104.09864}
}
@misc{glassllm_gqa_2023,
title = {GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints},
author = {Joshua Ainslie and James Lee-Thorp and Michiel de Jong and Yury Zemlyanskiy and Federico Lebrón and Sumit Sanghai},
year = {2023},
eprint = {2305.13245},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.2305.13245},
url = {https://arxiv.org/abs/2305.13245}
}
@misc{glassllm_prenorm_2020,
title = {On Layer Normalization in the Transformer Architecture},
author = {Ruibin Xiong and Yunchang Yang and Di He and Kai Zheng and Shuxin Zheng and Chen Xing and Huishuai Zhang and Yanyan Lan and Liwei Wang and Tie-Yan Liu},
year = {2020},
eprint = {2002.04745},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.2002.04745},
url = {https://arxiv.org/abs/2002.04745}
}
@misc{glassllm_residual_2015,
title = {Deep Residual Learning for Image Recognition},
author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},
year = {2015},
eprint = {1512.03385},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1512.03385},
url = {https://arxiv.org/abs/1512.03385}
}
@misc{glassllm_tied_2016,
title = {Using the Output Embedding to Improve Language Models},
author = {Ofir Press and Lior Wolf},
year = {2016},
eprint = {1608.05859},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1608.05859},
url = {https://arxiv.org/abs/1608.05859}
}
@misc{glassllm_autodiff_2015,
title = {Automatic differentiation in machine learning: a survey},
author = {Atilim Gunes Baydin and Barak A. Pearlmutter and Alexey Andreyevich Radul and Jeffrey Mark Siskind},
year = {2015},
eprint = {1502.05767},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1502.05767},
url = {https://arxiv.org/abs/1502.05767}
}
@misc{glassllm_adam_2014,
title = {Adam: A Method for Stochastic Optimization},
author = {Diederik P. Kingma and Jimmy Ba},
year = {2014},
eprint = {1412.6980},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.1412.6980},
url = {https://arxiv.org/abs/1412.6980}
}