Daniel & Friends / Glossary / Inference and latency
AI and automationInference and latency
Inference is the running of a finished model: it takes your input and produces an answer. Latency is the time until you get it, often measured to the first token and to the full response. The speed and price of inference decide where AI can be used in a product.
Daniel Votruba ·
In practice
For user-facing chat, watch time to first token above all. Batch jobs such as sorting reviews in bulk need less speed and cost less.
Related terms
- Token: A token is the unit of text a language model works with: usually a fragment of a word or a short word.
- Reasoning model: A reasoning model is a language model that spends time thinking before it answers: it breaks a task into steps, tries options and checks itself.
- Small language model (SLM): A small language model (SLM) is a language model with far fewer parameters than the largest systems.
- Large language model (LLM): A large language model is a model trained on a huge amount of text that predicts the next words and can write, summarise and answer.
More terms in the group AI and automation
Jailbreak (AI) · Human in the loop · Hallucination · Machine learning (ML) · Guardrails · MCP (Model Context Protocol) · Grounding · Model bias (AI bias) · Generative AI · Model evaluation (evals) · Function calling (tool use) · Multi-agent system · Foundation model · Multimodal model · Fine-tuning · Neural network · Few-shot and zero-shot prompting · NLP (natural language processing) · Explainable AI (XAI) · No-code and low-code · EU AI Act · Open-weight model · Embedding · Predictive analytics
Terms in practice
Need help putting them to work?
Terms are one thing, putting them to work is another. We go through which of them pay off for you and say so plainly. You get a first opinion for free and we reply within 24 hours.
Book a call