FundamentalsUpdated 2026-10-23
Latency
The time from sending a request to receiving the model response.
#性能#体验#工程
In plain language
Latency usually splits into time to first token and total generation time. Longer inputs, larger models and longer outputs increase it. Streaming reduces perceived latency. Real-time applications should watch P95, not just averages.
Example
A one-second first token and eight-second full answer combine into the user experience.
Recommended tools
Related terms
FundamentalsFeatured
Context Window
The maximum number of tokens a model can read and remember at once, including input and output.
3 related terms
Fundamentals
Inference
The computation a trained model performs to produce output from an input.
3 related terms
FundamentalsFeatured
Token
The smallest unit of text a model processes, often a word, subword or character group.
3 related terms
Applications
Agentic Workflow
Breaking a task into steps planned by AI, executed by tools and reviewed by humans.
3 related terms