Glossary
53 terms, each explained twice: once the way you would explain it to a friend, once the way it actually works. If a word on this site is unfamiliar, it should be here.
Apple
MetalApple's GPU API — the layer every Mac inference stack ultimately sits onMLXApple's native machine learning framework designed for Apple SiliconUnified Memory Architecture (UMA)Shared physical memory pool for CPU, GPU, and Neural Engine on Apple SiliconWired Memory Limit (macOS)The macOS system cap on maximum GPU-addressable unified RAM
Architecture
AttentionThe mechanism that lets a model weigh which earlier tokens matter for the next oneGrouped-Query Attention (GQA)Sharing Key-Value heads across multiple Query headsMixture of Experts (MoE)Sparse architecture activating only a fraction of weights per tokenMulti-Head Latent Attention (MLA)Low-rank latent KV compression pioneered by DeepSeekMulti-Token Prediction (MTP)Training models to predict several future tokens at onceSliding Window Attention (SWA)Restricting attention to a fixed local token window to save memory
Formats
AWQActivation-aware Weight Quantization preserving salient weightsBitNet b1.58 (1-bit LLMs)Ternary weight quantization {-1, 0, +1} removing matrix multiplicationGGUFUniversal binary format for quantized models in llama.cppGPTQSecond-order error compensation quantization for GPUsNVFP4 (Blackwell FP4)NVIDIA's hardware 4-bit floating point format with micro-block scalingQuantizationCompressing model weights from FP16 to 4-bit/8-bit precision
Fundamentals
Arithmetic IntensityThe ratio of compute work (FLOPs) to memory movement (bytes)DecodeThe memory-bandwidth-bound phase generating tokens one-by-oneLocal InferenceRunning AI models entirely on your own devicePerplexity (PPL)The standard metric for measuring language model accuracy and compression lossPrefillThe compute-bound phase where the AI reads your promptRoofline ModelThe fundamental formula that proves why decode is bandwidth-bound
Hardware
DGX SparkPalm-sized deskside AI supercomputer with 128GB unified memoryDGX StationNVIDIA's desk-side workstation line — a data-center GPU under your deskGB10 SuperchipThe Grace-Blackwell chip at the heart of DGX SparkMikroTik CRS504-4XQ-INThe budget 100GbE QSFP28 desktop switch for 2–4 DGX Spark clustersMikroTik CRS804-4C-IN4x 400G QSFP-DD high-density switch for ultra-fast local clustersMikroTik CRS812-8C-4XQ-INMulti-node 8x 400G + 4x 100G backbone switch for large workstation labsNVLink-C2CThe ultra-fast coherent interconnect inside Grace Blackwell superchipsRoCEv2 (RDMA over Converged Ethernet)Zero-copy network protocol bypassing CPU for direct GPU-to-GPU memory accessVRAMHigh-bandwidth memory on discrete graphics cards
Models
DeepSeek-R1 & V3Open-weight frontier reasoning and 671B Mixture-of-Experts architectureGLM-4 & GLM-4VZhipu AI's bilingual powerhouses with 128k context and multimodal visionGoogle Gemma 2Google's lightweight open models with logit soft-capping and sliding window attentionQwen 2.5 FamilyAlibaba's open-weight models dominating local coding, math, and general benchmarks
Performance
Continuous BatchingIteration-level request scheduling for LLM serversDFlash & DFlash-2Diffusion-based speculative decoding for parallel multi-token draftingFlashAttentionIO-aware exact attention algorithm eliminating memory bottlenecksKV CacheCached Key-Value tensors that speed up token generationMemory BandwidthHow fast your hardware can read model weights — the real speed limit on local inferencePagedAttentionVirtual memory pagination for the KV cachePrompt CachingReusing precomputed KV states for shared prefixes and system promptsSpeculative DecodingUsing a small model to draft answers and a big model to check themTokens per SecondThe headline speed number — and two very different numbers get called by this nameTTFTTime To First Token - How long until the AI starts speaking
Tools
DSparkDistributed execution & kernel compilation framework for DGX Spark clustersExLlamaV2 (EXL2)Variable bit-rate quantization engine for fastest GPU inferenceFlashInferUltra-fast customizable GPU kernel library for LLM servingllama.cppThe foundational C++ engine for running LLMs anywhereOllamaThe easiest way to run a local model — one command, sensible defaultsvLLMHigh-throughput serving engine with PagedAttention
Training
No term matches that.