How can i correctly use a CPU to perform inference of a quantized model

[neavo]

I made some attempts, such as:

self.model = AutoModelForTokenClassification.from_pretrained(
    "resource/kg_ner_gpu",
    device_map = "cpu",
    torch_dtype = torch.float16,
    quantization_config = BitsAndBytesConfig(
        load_in_4bit = True,
        load_in_8bit = False,
        bnb_4bit_quant_type = "nf4",
        bnb_4bit_compute_dtype = torch.float16,
    ),
    local_files_only = True,
    low_cpu_mem_usage = True,
)

self.classifier = pipeline(
    "token-classification",
    model = self.model,
    tokenizer = self.tokenizer,
    aggregation_strategy = "simple",
)

But the speed is very slow, so is there a correct code snippet as an example?

[gyanu2507]

load_in_4bit on device_map="cpu" is the slow path. bitsandbytes 4-bit kernels are CUDA. On CPU it either dequantizes to fp16 and runs a naive matmul, or it errors. That matches "8-bit works-ish, 4-bit crawls."

For CPU NER, skip bitsandbytes. Export to ONNX Runtime / OpenVINO, or load the model in plain fp16/fp32:

model = AutoModelForTokenClassification.from_pretrained(
    "resource/kg_ner_gpu",
    torch_dtype=torch.float32,
    device_map="cpu",
)

If you have a GPU, drop device_map="cpu" and keep the BitsAndBytesConfig. 4-bit is there to fit weights in VRAM, not to make CPU inference fast.

If the checkpoint was saved already quantized, load_in_4bit=True on CPU still will not give you the CUDA kernel speed. Use a CPU-oriented quant (GGUF via llama.cpp, or int8 ONNX) for that box.