THE INSTITUTE / RESEARCH TOPICS
Inference
What happens when a model runs—and how hardware, workload, and pricing affect cost.
Explore this topic
Inference
Choose a layer to understand the tools and buying questions.
The quick answer
Inference is using a trained AI model to produce an output from new input. When a model classifies a document, drafts a reply, or generates an image, it is performing inference.
The cost depends on how you run it. An API may charge by usage; a dedicated deployment may charge for reserved capacity; a self-hosted model consumes your own hardware and operating resources. A run uses resources even when you are not billed separately for that request.
What it is
Training changes the model’s learned parameters. Ordinary inference uses those parameters. A language model’s inference process commonly includes processing the input and generating output tokens. Context, model size, output length, and serving configuration can all affect the work required.
Know the difference
Tokens are not completed business work
A token bill describes one part of the system’s usage. A useful result may also require retrieval, tool calls, several model runs, validation, and human review.
Reasoning time is not a guarantee
Some systems spend more computation before answering. Whether that improves the result enough to justify the cost and delay must be tested on the task.
How it works
Prepare
The application gathers instructions and relevant input. Retrieval and data preparation can add time and cost before the model runs.
Generate
The serving system executes the model and returns its output. Longer requests and responses often require more work, but pricing also depends on the provider and service tier.
Accept or retry
The application or reviewer checks the result. Failed attempts, repeated tool calls, and retries belong in the workflow cost.
Types of tools
Platforms and tools
Business uses
Benefits and costs
Compare alternatives using the same workload and acceptance standard. Record time to a usable result, failure rate, review effort, model charges, and other operating costs. For dedicated capacity, include idle capacity and the team responsible for operating it.
A practical measure is total workflow cost divided by accepted results. Define the evaluation period and what you include in that total before comparing vendors.
Limits and risks
What the evidence shows
How we assess it
This guide combines primary documentation with the Institute’s questions for a buying team. It does not report a survey, independent benchmark, or vendor score.
Questions before you buy
- Are we paying for usage, reserved capacity, or our own deployment?
- What delay can this workflow tolerate?
- How long are typical inputs and outputs?
- How often do we retry or involve a person?
- What happens to cost and response time during peak demand?
- Which charges sit outside the model bill?
Common questions
Sources and reading
IBM — What is AI inference?: the basic distinction between training and inference.
IBM — What is LLM inference?: language-model execution.
OpenAI pricing documentation: an example of usage-based service pricing; verify current terms.
The cost framework is the Institute’s synthesis. No specific savings, throughput, or benchmark result is claimed.
What changed
October 24, 2026 — First publication as part of the Models research category.