What is inference?

1 of 6

How inference works
Prompt in. Tokens processed. Output out.
You send a prompt: You give the model a prompt, question, image, or document. This can be a simple question or a detailed request.
Text prompt
Question
Image
Document
Model reads as tokens: The model breaks your input into tokens and considers the available context, including your instructions, the content, and any relevant history.
Input
These | are | tokens | ...
Broken into tokens
Predicts the next token: The model uses what it knows and the current context to predict the next token, one step at a time. Each prediction becomes part of the response and helps predict the next one.
Model considers context
option | options | optionally
Predicts the next token
options
Tokens become an answer: The sequence of predicted tokens becomes the final output, such as a reply, summary, classification, or action.
Final output
Reply to your question
Summary of information
Classification or label
Action or next steps
Inference is a live prediction process, not new training.

Inference vs training
One builds the model. The other uses it.
Training
Goal: Learns from huge datasets to build model capabilities.
Data used: Uses large, diverse datasets (e.g. text, images, code).
What changes: Updates the model weights based on what it learns.
Who pays for it: Usually done by model builders or fine-tuners.
Inference
Goal: Answers live requests from users in real time.
Data used: Uses the already-trained model. No new training data.
What changes: Does not change the model weights.
Who pays for it: Happens every time a business user sends a task.
Most businesses mostly use inference, not training.

What affects inference?
Speed, cost, and quality change based on a few key inputs.
Model choice: Larger and more capable models often cost more and can be slower because they have more parameters to run.
Context size: Bigger prompts and more documents increase the amount of text the model has to process, which uses more compute and can take more time.
Output length: Longer answers require more output tokens, which increases latency and cost.
Tool calls or retrieval: Using tools, external data, or retrieval can add extra steps, which takes more time and may increase cost, but it often makes answers more accurate and useful.
The right setup makes inference faster, cheaper, and more useful.

A small-business example
A contractor uses inference to respond faster.
A contractor gets an email from a potential customer asking for a service appointment and a rough quote.
The AI reads the message, understands what is needed, drafts a reply, summarizes the job details, and suggests the next steps. This is inference because the model is using what it knows to produce a helpful response.
Customer email: "Hi, I need to schedule a visit to repair a leaking kitchen faucet. Can you tell me your availability and a rough estimate? I'm in Denver, CO."
AI uses inference: The model reads the email, understands the request, applies what it learned about scheduling, pricing, and customer communication, and generates a response based on that knowledge.
AI response and summary: "Hi! I can schedule a visit this Thursday or Friday. A faucet repair typically costs $150 to $250. Does either time work for you?"
Job details (from email):
Repair leaking kitchen faucet
Needs a service appointment
Location: Denver, CO
Rough estimate: $150 to $250
Suggested next steps:
Confirm preferred date and time
Share service area details
Send calendar link to book
Key benefits
Faster response time: Helps customers sooner.
More consistent replies: Uses the same tone and key information.
Less admin work: Summarizes details and suggests next steps automatically.
Inference is the engine behind everyday AI tasks.

The small-business takeaway
Inference is the moment AI goes to work.
Every AI request you send is inference: When you ask a question or give a task, the model uses what it already knows to generate a response.
Inference uses the trained model without retraining it: The model applies patterns it already learned to produce a new output.
Cost and speed depend on model choice, context size, and output length: Larger or more complex requests use more tokens, which can take longer and cost more.
Better prompts and focused context improve results: Clear instructions and only the relevant information help the model give you a higher quality answer.
Measure quality, latency, and cost before scaling: Test results, track performance, and understand costs so you can use AI confidently in your business.
Smart inference design turns AI into a useful business tool.

Lesson done.

Up next

What is temperature?

Where does this fit your business? Get your AI plan

Swipe to see each slide.

The short answer

Inference is the moment a trained AI model takes your request and produces an answer, using patterns it already learned rather than learning anything new. Every question you send to an AI tool is inference, and its speed and cost depend on the model chosen, how much text you send and how long the answer is.

Three things to remember

01Training builds the model, while inference is the model answering live without changing itself.
02The model breaks your input into small pieces and predicts its answer one piece at a time.
03Bigger models, longer prompts and longer answers often cost more and run slower, so measure cost and speed early.

Every slide, in words

01
What is inference?
When a trained model uses what it knows to produce an answer.
Inference is the moment an AI model takes your prompt, reads the input, and generates an output. It is the live use of the model after training is already done.
The model is not learning from scratch in that moment. It is applying patterns it already learned to predict the next best output.
User prompt: You give the model a question or task.
Trained model: The model uses what it learned during training.
Answer: The model generates a response based on your input.
Key idea
Training teaches the model.
Inference is the model answering now.
Every AI request you send is inference.
Training builds the brain. Inference puts it to work.
02
How inference works
Prompt in. Tokens processed. Output out.
You send a prompt: You give the model a prompt, question, image, or document. This can be a simple question or a detailed request.
Text prompt
Question
Image
Document
Model reads as tokens: The model breaks your input into tokens and considers the available context, including your instructions, the content, and any relevant history.
Input
These | are | tokens | ...
Broken into tokens
Predicts the next token: The model uses what it knows and the current context to predict the next token, one step at a time. Each prediction becomes part of the response and helps predict the next one.
Model considers context
option | options | optionally
Predicts the next token
options
Tokens become an answer: The sequence of predicted tokens becomes the final output, such as a reply, summary, classification, or action.
Final output
Reply to your question
Summary of information
Classification or label
Action or next steps
Inference is a live prediction process, not new training.
03
Inference vs training
One builds the model. The other uses it.
Training
Goal: Learns from huge datasets to build model capabilities.
Data used: Uses large, diverse datasets (e.g. text, images, code).
What changes: Updates the model weights based on what it learns.
Who pays for it: Usually done by model builders or fine-tuners.
Inference
Goal: Answers live requests from users in real time.
Data used: Uses the already-trained model. No new training data.
What changes: Does not change the model weights.
Who pays for it: Happens every time a business user sends a task.
Most businesses mostly use inference, not training.
04
What affects inference?
Speed, cost, and quality change based on a few key inputs.
Model choice: Larger and more capable models often cost more and can be slower because they have more parameters to run.
Context size: Bigger prompts and more documents increase the amount of text the model has to process, which uses more compute and can take more time.
Output length: Longer answers require more output tokens, which increases latency and cost.
Tool calls or retrieval: Using tools, external data, or retrieval can add extra steps, which takes more time and may increase cost, but it often makes answers more accurate and useful.
The right setup makes inference faster, cheaper, and more useful.
05
A small-business example
A contractor uses inference to respond faster.
A contractor gets an email from a potential customer asking for a service appointment and a rough quote.
The AI reads the message, understands what is needed, drafts a reply, summarizes the job details, and suggests the next steps. This is inference because the model is using what it knows to produce a helpful response.
Customer email: "Hi, I need to schedule a visit to repair a leaking kitchen faucet. Can you tell me your availability and a rough estimate? I'm in Denver, CO."
AI uses inference: The model reads the email, understands the request, applies what it learned about scheduling, pricing, and customer communication, and generates a response based on that knowledge.
AI response and summary: "Hi! I can schedule a visit this Thursday or Friday. A faucet repair typically costs $150 to $250. Does either time work for you?"
Job details (from email):
Repair leaking kitchen faucet
Needs a service appointment
Location: Denver, CO
Rough estimate: $150 to $250
Suggested next steps:
Confirm preferred date and time
Share service area details
Send calendar link to book
Key benefits
Faster response time: Helps customers sooner.
More consistent replies: Uses the same tone and key information.
Less admin work: Summarizes details and suggests next steps automatically.
Inference is the engine behind everyday AI tasks.
06
The small-business takeaway
Inference is the moment AI goes to work.
Every AI request you send is inference: When you ask a question or give a task, the model uses what it already knows to generate a response.
Inference uses the trained model without retraining it: The model applies patterns it already learned to produce a new output.
Cost and speed depend on model choice, context size, and output length: Larger or more complex requests use more tokens, which can take longer and cost more.
Better prompts and focused context improve results: Clear instructions and only the relevant information help the model give you a higher quality answer.
Measure quality, latency, and cost before scaling: Test results, track performance, and understand costs so you can use AI confidently in your business.
Smart inference design turns AI into a useful business tool.

More in part 2

Next step

Where does this fit your business?

Six quick questions about your business, then your AI plan: what to start with, what a person still checks, and the lessons that go with it. Or bring it straight to a free thirty-minute call.

Get your AI plan Book a free call