Transformers adds support for running GGUF (llama.cpp) quantized models
Hugging Face added support in transformers to run GGUF models (llama.cpp quantized checkpoints) locally via from_pretrained and kernels, with initial focus on Apple Silicon and the Qwen3.5 architecture.
What changed
Transformers can load and run GGUF checkpoints (example: unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf) using ggml/Metal kernels and the standard from_pretrained API.
Why it matters
Developers on Apple Silicon Macs can run, evaluate, and serve quantized GGUF checkpoints in Python/PyTorch via transformers, enabling local generation and testing workflows.
Who should care
DevelopersResearchers
Worth trying
Yes — Requires an Apple Silicon Mac, transformers (main) and compatible kernels.