A GPU is a chip that was born to draw graphics and does thousands of simple calculations at the same time. AI uses it because a model is, mostly, multiplying huge tables of numbers. Its edge is not speeding up a single operation, but running a huge number of them in parallel.
A question from the app. Pick an answer.
Why does AI usually run on GPUs?
A model does mountains of multiplications, and a GPU does thousands in parallel. A small model can also run on a CPU, just slower.
Its edge is not speeding up any single operation: it is running thousands in parallel, like a model's multiplications.
The GPU understands nothing: it does math. What the model "knows" is in its weights, which are numbers.
To draw a game frame, the GPU works out the color of millions of pixels, and each one is a small, independent calculation. That is why it has thousands of simple cores instead of a few powerful ones like a CPU.
A neural network has the same kind of work: multiplying matrices, that is, tables of numbers, over and over. Today's language models are Transformers, a design built to take advantage of doing those calculations in parallel.
Pretraining a huge model from scratch takes a lot of specialized compute: large groups of GPUs, or of similar chips like Google's TPUs, working together. After that, every answer is computing too: the model does math for every token it writes, for every user. That is the difference between training and inference.
That is why these chips are expensive, scarce and in high demand, and why using AI is not free: every answer takes up hardware and electricity. More in how much AI costs.
A dedicated GPU has its own memory, VRAM; with unified memory, as on Apple Silicon Macs, the CPU and GPU share it. The weights and what the model uses while running have to fit there. If they do not fit, it runs slowly or will not start. That is why quantization exists, to shrink the model, and why it is the first question when you run a local model.
"No GPU, no AI." A small model runs on a CPU, just slower, and many phones have their own AI chip, the NPU: that is on-device AI. "The GPU does the thinking." The GPU only does math; what the model knows is in its weights. "A GPU is faster at everything." Its strength is running many operations in parallel, not speeding up any single operation.