Local inference engines and model selection
View Full Diagram

I'm learning the best tools and models to use for local inference. It turns out there are several things you have to take into account. Since I'm no expert, I asked Perplexity to help me out. Most text here is directly from Perplexity so consider that.

Models and Engines

This is all about running local models. The research on LLMs is on fire. Everything changes (gets better) week-to-week. When you run foundation models that are served only by the vendor with their proprietary code, they can do all kinds of crazy stuff that may not be supported by particular local models and the inference engines. When you are running models locally, there is a distinct difference between what is in a downloadable model and how they work with available inference engines.

Rules For Selecting Local Inference Components

  1. Specify the hardware you are using
  2. Decide whether you need speed of output, depth of reasoning, best coding capability, etc.
  3. Choose a model that will fit your requirements
  4. Determine which inference engines can use that model
  5. Benchmark two or three viable engines with a real workload (and verify whether they work on your hardware).
  6. Select the engine that works best for your requirements

Quick Comparison

Format Primary Runtime Typical Packaging Primary Use Case Portability
GGUF llama.cpp and GGUF-compatible apps One .gguf file Quantized local inference, CPU use, GPU offload High
Hugging Face SafeTensors Transformers, vLLM, SGLang, PyTorch Weight files plus configs/tokenizer assets Native releases, serving, fine-tuning, newest models High across ML frameworks
MLX / MLX-LM checkpoints MLX and MLX-LM HF-like model directory, often with SafeTensors Efficient inference and fine-tuning on Apple Silicon Primarily Apple Silicon

Engines: vLLM, llama.cpp, Hugging Face Transformers (Python)

  • vLLM is the best fit when you want high-throughput, GPU-centric serving of native Hugging Face checkpoints—especially multiple concurrent requests, long contexts, efficient KV-cache use, batching, and direct Python integration—provided its support for the exact architecture and your CUDA/ROCm stack is mature.
  • llama.cpp is the pragmatic local-runtime choice for GGUF models, strong quantization, CPU inference, partial GPU offload, broad hardware portability, and simple single-user interactive use, but it can lag new model architectures and does not execute a model’s original HF/Python implementation.
  • Hugging Face Transformers in Python is the compatibility and experimentation baseline, giving you the most direct access to original safetensors checkpoints, tokenizers, processors, custom architectures, training/fine-tuning tooling, and reference behavior, but it generally takes more engineering to achieve vLLM-class serving throughput and may require trust_remote_code for models that ship custom repository code.

LLM Model File Formats

  • GGML Universal File Format: GGUF (.gguf)
  • A self-contained binary model format used primarily by the llama.cpp / GGML ecosystem.
  • Usually contains quantized model weights, architecture metadata, tokenizer data, special tokens, and other information a compatible GGUF runtime needs to load the model.
  • Commonly distributed as one file, for example: Qwen3-32B-Q4_K_M.gguf.
  • Supports many quantization types, such as Q4, Q5, Q6, Q8, IQ4, and others. These reduce RAM and VRAM requirements compared with FP16 or BF16 weights.
  • Best suited to local inference, including CPU inference, limited-VRAM GPUs, partial CPU/GPU layer offload, and portable deployment across NVIDIA, AMD, Apple Silicon, and other supported backends.
  • Used by tools such as llama.cpp, Ollama, LM Studio, KoboldCpp, Jan, and other GGUF-compatible applications.
  • Contains model data and declarative metadata, not arbitrary executable model code, CUDA kernels, Python source, or the original Hugging Face model implementation.
  • The inference engine supplies the actual forward-pass implementation, compute kernels, KV-cache logic, sampling, speculative decoding, and hardware acceleration.

  • Tradeoff: a model must be explicitly supported by the installed version of llama.cpp or another GGUF runtime. New architectures or special model features can take time to reach GGUF-compatible engines.

  • Hugging Face SafeTensors (.safetensors)

  • A safe binary tensor serialization format commonly used for the original, native release of modern Hugging Face models.
  • Stores model tensors without using Python pickle, avoiding the arbitrary-code-execution risk associated with loading untrusted pickled PyTorch checkpoint files.
  • Supports efficient memory-mapped or zero-copy-oriented loading behavior where the framework and operating system support it.
  • Normally appears as part of a model directory rather than as a single standalone runnable file.

  • A typical Hugging Face model directory may include:

    text model-00001-of-00004.safetensors model-00002-of-00004.safetensors model-00003-of-00004.safetensors model-00004-of-00004.safetensors model.safetensors.index.json config.json tokenizer.json tokenizer_config.json special_tokens_map.json generation_config.json chat_template.jinja

  • The SafeTensors files generally contain weights only; configuration, architecture information, tokenizer assets, generation defaults, and prompt templates usually live in neighboring JSON or text files.

  • Can store weights in BF16, FP16, FP32, FP8, INT8, or quantized checkpoint layouts such as GPTQ or AWQ, depending on the model release.
  • Used by Hugging Face Transformers, vLLM, SGLang, Text Generation Inference, PyTorch-based tools, fine-tuning frameworks, and many vendor inference runtimes.
  • Usually the best format for new model architectures, fine-tuning, multimodal models, GPU serving, and feature-complete reproduction of the original model release.

  • Some Hugging Face repositories include additional Python files such as modeling_*.py or configuration_*.py. Those files are separate from SafeTensors; frameworks only execute them when configured to use remote/custom code, such as via trust_remote_code=True.

  • MLX / MLX-LM Model Checkpoints

  • A model packaging and runtime ecosystem for MLX, Apple’s array and machine-learning framework for Apple Silicon.
  • Primarily intended for efficient local inference and fine-tuning on M-series Macs, using Apple’s unified memory architecture and Metal GPU acceleration.
  • “MLX format” is usually not one universal file extension comparable to .gguf; MLX-LM repositories often use a Hugging Face-like directory structure with model configuration, tokenizer files, and frequently *.safetensors weights.
  • MLX-compatible models may be converted or arranged differently from the original Hugging Face release so MLX-LM can load them efficiently.
  • MLX-LM releases may include MLX-specific quantization metadata and low-bit weight layouts optimized for MLX and Apple GPU kernels.

  • A typical MLX-LM-compatible repository may resemble:

    text config.json tokenizer.json tokenizer_config.json model.safetensors quantization_config.json generation_config.json

  • Used mainly by mlx-lm, MLX Python applications, and Apple-Silicon-oriented desktop tools.

  • Best suited to Apple Silicon users who want a native Python workflow, efficient unified-memory use, and strong local performance on a Mac.
  • Tradeoff: MLX checkpoints are substantially less portable than GGUF and have a smaller inference-engine ecosystem than generic Hugging Face SafeTensors checkpoints.