A guide to key AI LLM concepts: KV cache, weights and quantization

As hardware for running local AI models continues to advance, a new set of terminology is emerging. This primer explains crucial concepts to help developers and enthusiasts understand how large language models function under the hood.

• A model is an LLM computed from massive text training sets using token statistics.
• Weights represent the connection strength between tokens and determine output relationships.
• Quantization cleverly rounds weights to reduce model size for consumer hardware compatibility.
• The context window acts as working memory for storing session history, prompts, and files.

Understanding Models and Weights

A large language model is roughly analogous to advanced statistical analysis layers operating on tokens like words, code, and symbols. These computing layers require corresponding weights to function, representing how strongly connected different token sections are based on training data.

When viewing model specs like 4B or 70B parameters, it refers to the volume of these weights. Open-weight models allow users to tweak parameters freely on local hardware, providing deep customization options for developers who have the right processing power.

Quantization and Precision Levels

Memory is a finite resource, especially within VRAM-limited GPUs, driving developers to use the smallest data types possible. While AI labs use high precision during training, final versions undergo quantization to reduce model size and speed up inference.

Lower precision levels, such as INT4 or INT8, allow large models to run smoothly on consumer hardware like Mac Studio or desktop GPUs by trading off a fraction of response accuracy for significantly reduced storage requirements.

Context Windows and Memory Limits

The context window serves as the LLM's thinking space, holding everything associated with a session, including system prompts, chat history, and uploaded files. All these distinct concepts are reduced to a single pile of tokens for the model to process.

While larger context windows allow for extended conversations, extremely large windows can trigger the 'lost in the middle' effect, where the model focuses on the beginning and end of the session while overlooking details in the middle.