Field manual

Local AI, plainly explained.

Every term the radar uses, answered in one breath first and a minute of plain language second — no hype, no prior knowledge assumed. Each entry ends where the concept does real work on this site.

21 terms and growing

Also in:Español Português

01Context WindowThe context window is how much text a model can consider at once, measured in tokens — and holding that text costs memory through the KV cache.02DistillationDistillation trains a small model to imitate a larger one's answers — the reason many of the best small models exist, and why they carry a bigger model's habits, good and bad.03EmbeddingAn embedding is a list of numbers that captures the meaning of a piece of text, so that texts about the same thing end up close together — the machinery behind semantic search and RAG.04Fine-TuningFine-tuning continues a model's training on your own examples so it adopts a style, format or specialty — it changes the weights, where a prompt merely instructs them.05GGUFGGUF is the single-file format local AI runtimes load: a model's weights, usually quantized, packed together with everything needed to run it.06InferenceInference is a model answering — using its already-trained weights to produce output. Training is how those weights were learned; inference is every use after that.07KV CacheThe KV cache is the working memory a model fills while reading your conversation — it grows with every token, and at long contexts it can rival the model itself in size.08Local RuntimeA local runtime is the app that actually runs models on your machine — it loads a GGUF file, drives your GPU, and gives you a chat window or an API.09Mixture of Experts (MoE)A Mixture-of-Experts model keeps many specialist sub-networks but activates only a few per token — big-model knowledge at small-model speed, with big-model memory still required.10OffloadingOffloading splits a model between graphics memory and ordinary RAM when it does not fully fit — the model runs, just noticeably slower.11Parameters (7B, 70B)Parameters are a model's learned weights, counted in billions — the B in 7B. More parameters mean more memory, and only sometimes better answers.12QuantizationQuantization stores a model's weights at lower precision — for example 4-bit instead of 16-bit — cutting memory to roughly a quarter at a small quality cost.13RAG (retrieval-augmented generation)RAG lets a model answer from your own documents by fetching the relevant passages first and placing them into the prompt — knowledge without retraining anything.14SafetensorsSafetensors is the format models are originally published in: full-precision weights across many files, built for data-center GPUs and training — not for laptops.15Sampling (top-p, top-k)Sampling is how a model picks each next token from its list of candidates: temperature reshapes the odds, top-p and top-k trim the list, and together they set how predictable or adventurous the output is.16System promptA system prompt is the standing instruction a model reads before your conversation: who it should be, what it may do, and how to answer — set by the app or by you, not by the model.17TemperatureTemperature is the dial that sets how adventurous a model's word choices are: low values make it predictable and repetitive, high values creative and more error-prone.18TokenA token is the unit AI models actually read and write — a word fragment of a few characters, with roughly 750 English words fitting into 1,000 tokens.19Trending ScoreThe trending score is Hugging Face's own measure of which models the community is engaging with right now — a momentum signal, not a quality grade.20Unified MemoryUnified memory is one pool of RAM shared by the processor and the graphics cores — the reason Apple Silicon Macs can run models that would need an enormous graphics card elsewhere.21VRAM vs. System RAMVRAM is the memory on your graphics card, system RAM the memory on the motherboard. For local AI the difference is speed, not size: a model must mostly fit into VRAM to run fast.