What is latency?

1 of 6

Where latency comes from
Every step in the chain adds a little time
Think: many tiny delays add up.
Network travel: Requests and responses still need to move between the user, the app, and the model.
Queueing and load: Busy systems may make requests wait before processing begins.
Model thinking time: More complex reasoning usually takes longer than a simple answer.
Tool and data calls: Looking up files, databases, or external tools can add extra delay.
Output length: Longer answers, code blocks, or reports usually take longer to generate.
Latency usually comes from the whole workflow, not just the model alone.

Low vs. high latency
The right speed depends on the job
Same concept. Different user experience.
Low latency
Live chat: People expect a quick reply.
Search and Q&A: Users are actively waiting.
Copilots in the workflow: Delay breaks momentum.
Voice experiences: Lag feels awkward fast.
Higher latency can be OK
Nightly reports: Nobody is waiting live.
Document cleanup: Bulk work can run later.
Back-office tagging: Speed matters less than throughput.
Scheduled summaries: A delay is acceptable if cost is lower.
Good latency is not always the fastest latency: it is the right speed for the task.

What affects latency?
Model choice, tools, and design all play a role
The biggest levers.
Model size and complexity: More powerful models may take longer than smaller, faster ones.
Prompt and context length: Long instructions and large attached context increase work.
Extra tool calls: Retrieval, APIs, and database lookups can add more seconds.
Infrastructure and traffic: Server location, congestion, and system load all matter.
Streaming and caching: Good app design can make responses feel faster to the user.
Latency is often improved by better workflow design, not just by swapping models.

When latency matters most
Speed matters most when a person is waiting
User-facing moments need faster answers.
Customer support: Long pauses make service feel weaker.
Website search: Slow answers hurt conversion and trust.
Live copilots: The assistant must keep up with the user.
Voice and phone agents: Delay makes conversation feel unnatural.
Critical workflows: Approvals, alerts, and fraud review often need fast response.
If a delay is obvious to the customer or employee, latency is a business issue.

Small business takeaways
Measure, prioritize, and improve what users feel
A practical way to manage latency.
Measure the full wait time: Track the delay from click to final answer, not just model speed.
Match the model to the job: Use fast models for simple tasks and stronger ones where quality matters.
Keep prompts and context lean: Only send the instructions and data the task truly needs.
Use batch for non-urgent work: Move slow background tasks off the live path when possible.
Design for perceived speed: Streaming, status updates, and good UX can make waiting feel shorter.
The goal is not zero latency: it is the right balance of speed, cost, and quality.

Lesson done.

Up next

Open weights vs. closed models

Where does this fit your business? Get your AI plan

Swipe to see each slide.

The short answer

Latency is the delay between sending a request to an AI system and getting the response back: the wait you feel. It matters most when a person is waiting, such as live chat or phone agents, and matters much less for background work like nightly reports.

Three things to remember

01Delay comes from the whole workflow: network, busy systems, model thinking time, data lookups and answer length.
02Keep prompts lean, match the model to the job, and run slow, non-urgent work in batches.
03Measure the full wait from click to final answer, not just the model's speed.

Every slide, in words

01
What is latency?
Latency is the delay between a request and the response
Think: wait time.
A request is sent: A user clicks, asks, or uploads something and the system starts working.
The system does the work: Data moves through the model, tools, and infrastructure before an answer comes back.
The response arrives: The total delay you feel is latency, whether it lasts milliseconds or seconds.
Latency is not what the AI knows: it is how long the answer takes to show up.
02
Where latency comes from
Every step in the chain adds a little time
Think: many tiny delays add up.
Network travel: Requests and responses still need to move between the user, the app, and the model.
Queueing and load: Busy systems may make requests wait before processing begins.
Model thinking time: More complex reasoning usually takes longer than a simple answer.
Tool and data calls: Looking up files, databases, or external tools can add extra delay.
Output length: Longer answers, code blocks, or reports usually take longer to generate.
Latency usually comes from the whole workflow, not just the model alone.
03
Low vs. high latency
The right speed depends on the job
Same concept. Different user experience.
Low latency
Live chat: People expect a quick reply.
Search and Q&A: Users are actively waiting.
Copilots in the workflow: Delay breaks momentum.
Voice experiences: Lag feels awkward fast.
Higher latency can be OK
Nightly reports: Nobody is waiting live.
Document cleanup: Bulk work can run later.
Back-office tagging: Speed matters less than throughput.
Scheduled summaries: A delay is acceptable if cost is lower.
Good latency is not always the fastest latency: it is the right speed for the task.
04
What affects latency?
Model choice, tools, and design all play a role
The biggest levers.
Model size and complexity: More powerful models may take longer than smaller, faster ones.
Prompt and context length: Long instructions and large attached context increase work.
Extra tool calls: Retrieval, APIs, and database lookups can add more seconds.
Infrastructure and traffic: Server location, congestion, and system load all matter.
Streaming and caching: Good app design can make responses feel faster to the user.
Latency is often improved by better workflow design, not just by swapping models.
05
When latency matters most
Speed matters most when a person is waiting
User-facing moments need faster answers.
Customer support: Long pauses make service feel weaker.
Website search: Slow answers hurt conversion and trust.
Live copilots: The assistant must keep up with the user.
Voice and phone agents: Delay makes conversation feel unnatural.
Critical workflows: Approvals, alerts, and fraud review often need fast response.
If a delay is obvious to the customer or employee, latency is a business issue.
06
Small business takeaways
Measure, prioritize, and improve what users feel
A practical way to manage latency.
Measure the full wait time: Track the delay from click to final answer, not just model speed.
Match the model to the job: Use fast models for simple tasks and stronger ones where quality matters.
Keep prompts and context lean: Only send the instructions and data the task truly needs.
Use batch for non-urgent work: Move slow background tasks off the live path when possible.
Design for perceived speed: Streaming, status updates, and good UX can make waiting feel shorter.
The goal is not zero latency: it is the right balance of speed, cost, and quality.

More in part 8

Next step

Where does this fit your business?

Six quick questions about your business, then your AI plan: what to start with, what a person still checks, and the lessons that go with it. Or bring it straight to a free thirty-minute call.

Get your AI plan Book a free call