Batch processing, or the batch API, is an option in some AI APIs where you send many requests together and the provider runs them when it has spare capacity. Results come back in hours, not instantly, and in exchange it costs less. It suits big jobs where nobody is waiting for the answer.
A question from the app. Pick an answer.
What job is batch mode for?
Batch does not answer instantly, and a live chat needs the answer now.
Batch charges less in exchange for not answering instantly. It works if you can wait; check the service's turnaround.
Right now is the opposite of batch: it does not answer instantly. The savings are paid for with waiting.
Instead of sending one request and waiting for the answer, you prepare a file with all the requests and send it in a single call to the API. The provider processes it in the background, when its servers are less busy, and later you download the results. The provider usually sets a maximum window, such as 24 hours, though it often finishes sooner.
Made up rates: normal input costs 1 USD per million tokens and in batch mode, 0.50 USD. You want to tag 50,000 reviews of about 200 tokens each: that is 10 million input tokens. In normal mode, 10 USD; in batch, 5 USD. Output is cheaper too in this example. Real discounts are set by each provider, and may not apply to every model.
It is savings paid for with waiting. It can stack with prompt caching, but each provider decides whether the two can be combined, so check its rules.
It fits tagging reviews, summarizing thousands of documents, translating a catalog or testing a model against many examples: big work that can wait. It does not fit a live chat or anything that needs the answer right now, because the result can take hours. It is also a poor choice for an urgent request or one you will adjust halfway through.
Before sending the whole batch, try a few requests first: a mistake in the instructions repeats across all 50,000. To see everything that moves the bill, check the other page.