Transformers documentation

SpQR

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

SpQR

SpQR 量化算法采用了一种 16x16 分块的双层组 3 位量化结构,并包含稀疏离群值。

若要使用 SpQR 对模型进行量化,请参阅 Vahe1994/SpQR 仓库。

使用 from_pretrained() 加载 SpQR 量化模型。

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

quantized_model = AutoModelForCausalLM.from_pretrained(
    "elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf",
    dtype=torch.half,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf")
Update on GitHub