The explosion of open-source AI models has created an unexpected problem: choosing how to store and run them. Developers building with Hugging Face's model hub now face a bewildering array of formats, each with different trade-offs in speed, accuracy, and file size. Understanding these formats is no longer optional for anyone serious about deploying large language models (LLMs) locally.

The confusion starts with a fundamental misunderstanding. Most people conflate two separate concepts: how a model is stored on disk versus how its weights are compressed. A container format like GGUF or safetensors defines storage, while quantization methods like GPTQ and AWQ define compression. This distinction matters because the same quantization method can live in different containers, and different methods can achieve similar compression ratios through entirely different approaches.

What Are the Main Storage and Compression Formats?

The landscape includes several major players, each with distinct origins and use cases. Safetensors, created at Hugging Face, has become the security standard for uncompressed models. Unlike older PyTorch pickle files (.bin), which can execute arbitrary code when loaded, safetensors is a simple JSON header plus raw tensor buffers with nothing executable inside. This design eliminates a major security risk when downloading untrusted model checkpoints. Safetensors is now listed as a PyTorch Foundation project.

GGUF emerged as a specialized format for the llama.cpp ecosystem. Created by Georgi Gerganov, who also leads llama.cpp development, GGUF replaced an older format in August 2023. Unlike tensor-only formats, GGUF bundles the tokenizer, special tokens, and chat templates alongside weights in a single file. This self-contained approach makes deployment simpler and more portable.

The quantization methods represent different mathematical approaches to the same problem: how to squeeze model weights into fewer bits without destroying accuracy. GPTQ, developed by researchers at IST Austria and ETH Zurich, uses approximate second-order information to decide which weights can be rounded most aggressively. It appeared on arXiv in October 2022 and was published at ICLR 2023. The method quantized a 175-billion-parameter model in about 4 GPU hours down to 3 or 4 bits per weight with negligible accuracy loss.

AWQ (Activation-aware Weight Quantization) takes a different approach. Developed by Song Han's group at MIT and published in June 2023, AWQ won the MLSys 2024 Best Paper Award. Rather than treating all weights equally, AWQ identifies roughly 1% of "salient" weights that matter most for accuracy. Instead of storing these at higher precision, it scales them through a mathematically equivalent transformation, keeping a uniform hardware-friendly format. This approach requires no backpropagation or reconstruction, making it less likely to overfit to calibration data.

How Do These Formats Compare in Practice?

The practical differences matter for real deployments. A 7-billion-parameter model in full 16-bit precision requires about 13 gigabytes of memory. Using GGUF's Q8_0 quantization (8-bit, near-lossless) reduces this to 7 gigabytes with only a 0.03% increase in perplexity, a measure of prediction error. Pushing to Q4_K_M (4-bit with mixed precision) shrinks the model to 4.1 gigabytes but increases perplexity by 1.68%.

Calibration time differs significantly between methods. Hugging Face estimates that quantizing an 8-billion-parameter model with GPTQ takes about 20 minutes on a single NVIDIA A100 GPU. AWQ accomplishes the same task in roughly 10 minutes, nearly half the time. This speed advantage comes from AWQ's simpler mathematical approach.

Runtime performance also varies. In the original AWQ paper, the TinyChat runtime ran more than 3 times faster than Hugging Face's full-precision implementation on desktop and mobile GPUs. GPTQ reported end-to-end speedups of about 3.25 times on NVIDIA A100 hardware and 4.5 times on A6000 GPUs.

How to Choose the Right Format for Your Use Case

• For maximum compatibility and ease of deployment: Use GGUF with llama.cpp or Ollama. GGUF's self-contained design means you get the tokenizer and chat template in one file, and it works natively across the llama.cpp ecosystem including LM Studio, GPT4All, and Ollama. vLLM support exists but remains experimental and under-optimized.
• For production inference with established tooling: Choose GPTQ if you need integration with Transformers, vLLM, or SGLang. GPTQModel has fully supplanted the older AutoGPTQ library and works across these frameworks. GPTQ's group size parameter (typically 128) and act-order option let you fine-tune the accuracy-to-size trade-off.
• For fastest calibration and mobile deployment: Consider AWQ, which calibrates roughly twice as fast as GPTQ and delivers superior performance on edge devices. AutoAWQ is officially deprecated, but the format remains widely supported across inference libraries.

The GGUF format includes a labeling system that reveals compression details. A filename like "Q4_K_M.gguf" tells you the quantization scheme. The "K" variants use 256-weight super-blocks with 6-bit or 8-bit scales, while "I" variants (like IQ4_XS) use importance matrices computed from calibration data. Newer types include TQ1_0 and TQ2_0 for ternary weights, plus MXFP4, a 4-bit microscaling floating-point format.

A critical nuance: most GPTQ, AWQ, and other quantized models are also stored in safetensors files. The quantization lives in the tensor contents and configuration file, not in a new container. This means you can get the security benefits of safetensors while using any quantization method.

The choice between these formats ultimately depends on your infrastructure and constraints. If you're running models on consumer hardware with llama.cpp, GGUF offers the smoothest experience. If you're building production systems with vLLM or SGLang, GPTQ provides mature integration. If you're targeting mobile or edge devices, AWQ's speed advantage becomes compelling. Understanding these trade-offs transforms the format landscape from confusing to navigable.