Inference is the process of applying an already-trained AI model to new input in order to produce an output such as text, an image, or a classification.
While training is the expensive, one-off (or periodically repeated) process in which the model learns from data, inference is the actual productive use of the finished model – every single chat reply, image generation, or code completion is an inference operation.
Inference requires far less compute than training, but happens millions of times over once a model is in production use – for popular models, the cumulative cost of inference over time often exceeds the original training cost.
The speed and efficiency of inference (often measured in tokens per second) is a key competitive factor for AI model providers, since it directly affects both user experience and operating costs.
