Transformer
A neural network architecture built around self-attention and parallel training, the foundation of modern large models.
In plain language
A Transformer stacks self-attention, feed-forward layers, residual connections and normalization. Unlike recurrent networks, it sees the whole sequence at once, enabling GPU parallelism; positional encodings add order. Encoder-decoder designs suit translation, while decoder-only designs (the GPT family) suit generation. Scaling Transformers produced emergent capabilities but also memory, latency and training-cost challenges.
Example
ChatGPT, Claude and DeepSeek are all built on Transformer variants.
Recommended tools
Related terms
Attention Mechanism
A computation that lets a model dynamically weigh how relevant other positions are when processing one position.
Next-Token Prediction
The training objective of predicting the next token given the preceding text.
Positional Encoding
A mechanism that injects word-order information into a Transformer, since self-attention alone is order-agnostic.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.