Multimodal AI is AI that works with more than one kind of data: text, images, audio or video. It can take them in, produce them or both. You show it a photo of a whiteboard and it writes it up cleanly, or you talk to it and it answers. Many chat apps today already are, at least in part.
A question from the app. Pick an answer.
You paste 20 screenshots to an AI that reads images. How do they count in its context?
Images become tokens too and take up room in the context.
Each screenshot turns into tokens that go into the context. How many depends on the model and the size, and if you pay per use, you pay for them.
An image is not a word: it gets split into many pieces, and each one is a token.
Keep input and output apart. A model can understand a photo and answer only in text, without being able to draw anything. Another one generates images from what you type but cannot read them. A chat that "sees" your photos is not necessarily one that can make them.
Many apps bundle several pieces: an LLM that understands text and images, another model that draws and another that turns speech into text and text into speech. To you it is one chat; inside it can be three. The part that creates new content is generative AI; multimodal is about how many kinds of data it handles.
You snap a photo of a receipt and ask for the total. You photograph the whiteboard at the end of a meeting and get back the to-do list. Or you paste a screenshot of an error and it explains what went wrong.
The model does not look at the photo the way you do: it turns it into tokens, and those tokens take up room in the context window. How many depends on the model and the image size: one image can use hundreds or thousands. If you pay per use, they get billed. With audio, many apps turn it into text first, so the model only gets text.
"If it read it, it read it right." Small numbers, handwriting and blurry photos get mixed up easily: a 7 turns into a 1. Check the totals.
"It only says what it sees." If the photo is cropped, it can fill in what is missing in the same confident tone; that is a hallucination.
"An image cannot give orders." An image can carry instructions, even nearly invisible ones. If the model follows them as orders, they pull your request off course: that is a prompt injection.
"Multimodal means it takes anything." Some chats read photos but not audio, or cut long videos short. A PDF can hold text and images: the app accepting the file does not mean the model looks at the images. Check what yours does.