A Transformer is a kind of neural network architecture, introduced in 2017 in the paper "Attention Is All You Need". It is the basis of almost all of today's language models: the T in GPT comes from it. Its key piece is attention, which, for each token, weighs which other tokens matter and combines their information.
A question from the app. Pick an answer.
What is a Transformer in AI?
It was first shown on translation, but it is a general design: today it writes, summarizes and codes.
It is the design behind almost all of today's LLMs. The T in GPT comes from it.
A chip is hardware, like a GPU. The Transformer is the model's design, not the machine that runs it.
Read this sentence: "the dog did not cross the street because it was tired". What was tired? You know right away: the dog, not the street. A Transformer splits the text into tokens and, for each one, works out how much the others matter to it. That way attention helps relate parts of the text, such as "dog" and "tired".
Attention works with what the model has in view: its context window. When generating text, it looks at the earlier tokens, not the ones it has not written yet. It is not a memory that keeps your chats.
Before, many text neural networks read word by word, left to right, and each step waited for the one before. During training, the Transformer processes all the positions in the text in parallel. That is a huge number of operations at once, exactly what a GPU does best.
That made it possible to train on far more text and with much bigger models. That is where LLMs like GPT, Claude or Gemini came from. The idea is also used with images and audio.
"It is a translator." It was first shown on translation, but it is a general design: today it writes, summarizes and codes. "It is a chip." A chip is hardware; the Transformer is the model's design, not the machine that runs it. "All AI is a Transformer." There are other deep learning architectures, and hybrid models that mix attention with other ideas.