NLP Fundamentals
Transformer Architecture
The neural network architecture underlying every modern large language model — characterised by self-attention layers that let the model weigh relationships between all tokens in a sequence simultaneously.
Definition
The transformer is a neural network architecture introduced in 2017 in the paper 'Attention Is All You Need.' Its defining mechanism, self-attention, lets each token in a sequence consider its relationship to every other token in parallel, rather than processing the sequence one step at a time as earlier recurrent architectures did. This parallelism made it practical to train models on vastly larger datasets, which in turn produced the scaling behaviour that has driven progress in language modelling since. Every modern production LLM — GPT, Claude, Gemini, Llama — is a transformer variant.
Why it matters
Understanding that LLMs are transformers explains several of their characteristic behaviours. They handle long-range dependencies well within the context window because every token attends to every other. They are computationally expensive at long context because attention costs grow quadratically with sequence length — which is why a million-token window is a much bigger deal than it sounds. They have no inherent memory across calls because the architecture has no persistent state outside the context. For writers and product builders, the architecture is a reminder that the model is not a database, a search engine, or a reasoning engine — it is a pattern-matching machine over text sequences. Its strengths and weaknesses follow from that. It is excellent at continuing patterns it has seen. It is unreliable at tasks that require lookup, calculation, or grounded reasoning that the pattern does not encode.
Examples
Self-attention in practice
When the model generates the word 'it' in a sentence, attention layers weigh every preceding noun to determine the antecedent. This is how the model maintains reference across long sentences.
Quadratic cost
Doubling the context length quadruples the attention computation. This is why long-context models are expensive and why architectural innovations focus on attention efficiency.
No persistent memory
A 'memory' feature in a chat product is implemented as text retrieved into the context window. The architecture itself has no state between calls.
Frequently asked
Are all LLMs transformers?
Effectively all production LLMs today are transformers or close variants. Research into alternative architectures continues but none have displaced the transformer in production.
What is self-attention in one sentence?
A mechanism that lets every token in a sequence weigh its relationship to every other token, producing context-aware representations.
Why do transformers need so much compute?
Because attention scales with the square of sequence length and because useful capability emerges only at very large parameter counts trained on very large datasets.
Related terms
Large Language Model (LLM)
A neural network trained on very large text corpora to predict the next token in a sequence, capable of producing fluent natural-language output.
Context Window
The maximum amount of text — measured in tokens — that a language model can consider at once when generating its response, including both the prompt and the output.
Tokenization
The process of breaking text into the discrete units (tokens) a language model actually processes.
Embeddings
Numerical vector representations of text whose distances capture semantic similarity between pieces of content.
Put the concept to work
Open the rewrite engine and apply this principle to a draft of your own.
Try a rewrite