A local AI model is a model you download and run on your own computer, instead of using it on a company's servers. It can work offline, and your data can stay on your device if the app does not send it out. The tradeoff: what you can run depends on your hardware.
A question from the app. Pick an answer.
What resource does it need to hold the model while using it?
Holding the model requires memory on the device. Internet does not replace that resource.
Many models have weights available to download for free. Check their usage license.
It needs working memory, such as RAM or graphics memory. The amount depends on the model and its format.
When you use a chat like ChatGPT, your message travels to the company's servers, the model answers there and the reply comes back to your screen. With a local model, the model file (its weights) sits on your disk and a program on your computer does all the computing.
To use one you need two things: the downloaded model and a program that runs it, such as Ollama or LM Studio. With both installed, you can chat offline. Note: "local" is about where it runs, not where it was made or which language it speaks.
While you use it, the model has to fit in working memory: RAM or the graphics card's video memory. Disk space or fast internet are not enough. If it does not fit, it will not start or it will crawl.
An example calculation: an 8 billion parameter model, compressed to 4 bits per parameter, takes about 4 GB, plus some extra for the conversation. The same model uncompressed needs several times that. That is why many people use compressed versions: they fit on an ordinary laptop at the cost of a little quality.
When the data is sensitive: a clinic summarizing patient records or an accounting firm reviewing client statements. Also when the connection is unreliable or data is capped, which is common outside big cities: the model keeps working when the internet drops.
"Local guarantees privacy." It helps, but it depends on the software not sending your data out: check what the app does. "Local is free." Many models download at no cost, but they come with a usage license, and the hardware and electricity are on you. "Local is better (or worse) than cloud." It depends on the model, the task and your hardware.